help button home button Am J Pathol R & D Systems
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Purchase Article
Right arrow View Shopping Cart
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Szuhai, K.
Right arrow Articles by Raap, A. K.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Szuhai, K.
Right arrow Articles by Raap, A. K.
(American Journal of Pathology. 2001;159:1651-1660.)
© 2001 American Society for Investigative Pathology

Technical Advance

A Novel Strategy for Human Papillomavirus Detection and Genotyping with SybrGreen and Molecular Beacon Polymerase Chain Reaction

Károly Szuhai*, Emily Sandhaus*, Sandra M. Kolkman-Uljee{dagger}, Marc Lemaître{ddagger}, Jean-Christophe Truffert{ddagger}, Roeland W. Dirks*, Hans J. Tanke*, Gert Jan Fleuren{dagger}, Ed Schuuring{dagger} and Anton K. Raap*

From the Departments of Molecular Cell Biology*
and Pathology,{dagger}
Leiden University Medical Center, Leiden, The Netherlands; and the Department of Production,{ddagger}
Eurogentec S.A., Seraing, Belgium


   Abstract
Top
 Abstract
Introduction
 Materials and Methods
 Results
 Discussion
 References
 
Human papillomaviruses (HPVs) play an important role in the pathogenesis of cervical cancer. For identification of the large number of different HPV types found in (pre)malignant lesions, a robust methodology is needed that combines general HPV detection with HPV genotyping. We have developed for formaldehyde-fixed samples a strategy that, in a homogenous, real-time fluorescence polymerase chain reaction (PCR)-based assay, accomplishes general HPV detection by SybrGreen reporting of HPV-DNA amplicons, and genotyping of seven prevalent HPV types (HPV-6, -11, -16, -18, -31, -33, -45) by real-time molecular beacon PCR. The false-positive rate of the HPV SybrGreen-PCR was 4%, making it well suited as a prescreening, general HPV detection technology. The type specificity of the seven selected HPV molecular beacons was 100% and double infections were readily identified. The multiplexing capacity of the HPV molecular beacon PCR was analyzed and up to three differently labeled molecular beacons could be used in one PCR reaction without observing cross talk. The inherent quantitation capacities of real-time fluorescence PCR allowed the determination of average HPV copy number per cell. We conclude that the HPV SybrGreen-PCR in combination with the HPV molecular beacon PCR provides a robust, sensitive, and quantitative general HPV detection and genotyping methodology.


   Introduction
Top
 Abstract
 Introduction
Materials and Methods
 Results
 Discussion
 References
 
The association of human papillomaviruses (HPVs) with several human malignancies is well established.1 HPV has been found in benign, premalignant and malignant lesions of the skin2 and of the anogenital3 and aerodigestive system.4,5 World-wide epidemiological studies have shown that cancer of the uterine cervix is the second most common malignant disease in women.6 Virtually every cervical cancer (99.7%) is HPV-positive, indicating that the presence of HPV is an obligatory element in the development of cervical cancer.7

Until now more than 80 types of HPV have been identified. The types associated with diseases of the anogenital tract can be classified on the basis of phylogenetic relationship8 and of association frequencies with benign or malignant cervical lesions as high-risk types (HPV-16, -18, -31, -33, -35, -39, -45, -51, -52, -54, -56, -58, -59, and -66) and low-risk types (HPV-6, -11, -34, -40, -42, -43, -44). The most common HR types are HPV-16, -18, -31, -33, and -45.6,9 The most commonly found LR types are HPV-6 and -11.

Because the detection and type-specific classification of HPV infection by an in vitro viral culture test is not possible and serological tests are still ineffective, a molecular DNA diagnosis is needed. Direct hybridization-based assays, such as Southern blotting and in situ hybridization have been described, but these methods lack sensitivity and specificity. Signal amplification assays such as used in the hybrid capture assays10 for HPV are often applied. After recent improvements in this technique, it was shown that at the 1 pg/ml cut-off point (equivalent to 100.000 viral copies/ml) results agreed well with those of a polymerase chain reaction (PCR)-based assay.11

Because of the high-type specificity and sensitivity provided by target DNA amplification, the most widely used method for HPV detection in cervical cancers is based on PCR using either type-specific or general primers for the amplification.12-16 After the PCR amplification step, a number of methods can be used for classification based on sequence differences. These are based on type-specific oligonucleotide hybridizations such as for example the reverse line blot detection method, restriction fragment length polymorphism (RFLP), or, as the gold standard, sequencing.17-19 As a disadvantage in general, the above-mentioned methods are not suited for quantitative determination of viral load and require extensive processing after PCR.

The introduction of real-time reporting of PCR amplification with fluorogenic reporter probes such as TaqMan and molecular beacon probes provides quick and convenient tools for sequence variation detection (genotyping) and quantification.20,21 Here we designed molecular beacons recognizing common high-risk (HPV-16, -18, -31, -33, and -45) and low-risk (HPV-6, HPV-11) types and analyzed their performance in a general primed HPV-PCR in terms of specificity and sensitivity in a series of 53 formalin-fixed, paraffin-embedded cervical cancers that represented 21 different HPV types. Furthermore, we worked out a two-step HPV detection/genotyping strategy. In the first HPV step, samples are quantitatively analyzed for HPV-DNA using SybrGreen as a general HPV-amplicon reporter. In the second step, samples positive in the SybrGreen HPV-PCR are then analyzed for the presence of the seven prevalent HPV genotypes using the developed molecular beacon HPV-PCR. As internal reference, a ß-globin molecular beacon-PCR with identical PCR-cycling conditions as the HPV-PCR amplification is included in this strategy to assess DNA quality and to quantitate nuclear genome copy number of the samples.

This strategy, which is amenable to multiplexing, needs minimal amounts of tissue material and uses homogeneous assay formats that do not require processing after PCR, is well suited for quantitative detection of HPV-DNA, and for genotyping prevalent HPV types in cervical cancers.


   Materials and Methods
Top
 Abstract
 Introduction
 Materials and Methods
Results
 Discussion
 References
 
Samples

Cell Lines

Cell lines containing different types of integrated HPV were obtained from the American Type Culture Collection (ATCC, Rockville, MD) or established by us. Extracted HPV-16 positive DNA from the SiHa cell line (ATCC no. HTB35) or from the CaSki cell line (ATCC no. CRL 1550), HPV-18 DNA from the HeLa cell line (ATCC no. CCL2), and HPV-45 DNA from the CC-10A cell line22 were used for dilution series and as positive controls. A B-lymphoblastoid cell line not containing HPV DNA was used as a negative control. DNA was isolated from cell lines according to a standard proteinase K-SDS procedure.22

Patient Samples

All patient samples were retrieved from formaldehyde-fixed paraffin-embedded tissue blocks collected between 1991 to 2000 and archived at the Department of Pathology, Leiden University Medical Center. From each paraffin block, four sections were cut. The first and the fourth sections were of 5-µm thickness, mounted onto glass slides, and hematoxylin and eosin stained for histological evaluation. The second and third sections of 10-µm thickness were collected in 1.5-ml Eppendorf tubes for DNA isolation as described.23 To prevent cross-contamination, microtome blades were thoroughly cleaned and mock-tissue blocks (paraffin only) were cut between blocks.

For evaluation of the specificity of the HPV molecular beacon-PCR, 57 marked samples, representing 4 mock tissue and 53 HPV-positive tissue blocks of patients with squamous cell (n = 34), adenocarcinoma (n = 5), and adenosquamous (n = 2) carcinoma of the uterine cervix and CIN lesions (n = 10) were used, as well as HPV-positive biopsies from skin (n = 1) and vulva (n = 1). The original HPV typing was performed on basis of general primer (GP) HPV-PCR followed by sequencing.23 The HPV type was not known to the investigators performing the molecular beacon PCR (KS, ESa). The four mock tissue blocks were included as negative controls.

The 53 patient tissue blocks represent 21 different HPV types (HPV-6, -7, -11, -16, -18, -26, -31, -33, -35, -45, -51, -52, -53, -55, -56, -58, -59, -66, -68, -70, and -73). Of the 53 patient samples, 32 contained HPV types matching with the seven molecular beacon probes used here (HPV-6, -11, -16, -18, -31, -33, and -45), whereas 21 samples represented the 14 other types. It should be noted here, that this set of 21 tissue samples is biased toward the infrequent types.

For evaluation of the false-positive rate of the general HPV SybrGreen-PCR assay, tissue blocks of 25 randomly selected ovarian cancers were used, which are considered to be negative for HPV.24

To assess the feasibility of a two-step HPV detection strategy (general primer HPV-PCR amplification reporting by SybrGreen, followed by molecular beacon HPV-PCR typing). Nineteen samples derived from 8 cases representing carcinoma of the uterine cervix (6 cases), 1 CIN II lesion, and 1 laryngeal papilloma were additionally selected. Cervical cancer cases were sampled from different paraffin blocks containing the tumor, histologically tumor-free lymph nodes, and tumor-free operation edges. Again, neither HPV types nor sample origin was known to the investigators who performed the SybrGreen- and molecular beacon-PCRs.

General Primer-Mediated HPV-PCR and Sequencing

General primer-mediated PCR (GP-PCR) and subsequent sequencing was performed for the detection and typing of HPV DNA. The ß-globin gene was used as an internal control for PCR amplification with the RS40/RS42 primer pair resulting in a 188-bp amplicon.25 All samples were subjected to GP-PCR with the HPV-consensus CPIIG/CPI primer pair to amplify a 188-bp fragment in the highly conserved E1 ORF region,13 the MY09/MY11 primer pair to amplify a 450-bp fragment in the L1 open reading frame,14 and the GP5/6 primer pair to amplify a 152-bp fragment in the L1 open reading frame.15 To determine the HPV subtype, PCR products were subjected to direct sequence analysis. Briefly, after purification of the PCR products using a Microspin G50-column (Amersham Pharmacia Biotech, Treiburg, Germany), the products were sequenced directly with the Big Dye Terminator kit (Applied Biosystems) using the PCR primers and a ABI Prism 377 automated sequencer. For nucleotide sequence analysis and comparisons, the programs Seqed, Fasta, and MAP of the Wisconsin Genetics Computer Group (version 9.1) sequence analysis software package were used. All available sequences of HPV types were retrieved from GenBank and complemented with sequences stored in a local HPV nucleotide sequence database. HPV types were identified when nucleotide comparisons revealed an identity of more than 95% with a known type.

Design and Characterization of Molecular Beacons

Molecular beacon probes were designed following the guidelines described by Tyagi and Kramer21 and in the related Internet-site (www.phri.nyu.edu/molecular_beacons). For a given HPV type the probe sequence was chosen to have maximum sequence difference with all other HPV types and with a melting temperature between 50 to 55°C. The intramolecular configuration of molecular beacons and the amplicon probing was modeled with the DNA mfold program that uses thermodynamic parameters established by SantaLucia.26 Sequences resulting in earlier opening of the stem because of an unexpected intramolecular configuration were excluded. Selected molecular beacon probes were further tested for absence of primer-probe, and probe-probe cross-hybridization.

Molecular beacon sequences (Table 1) were synthesized using standard phosphoramidite chemistry and purified by high pressure liquid chromatography as described before.27 Briefly, molecular beacons were synthesized automatically with the 3'-DABSYL quencher linked to the solid phase support (DABSYL-CPG, Glen Research). Oligonucleotide chain elongation was run via standard cyanoethyl-phosphoramidite chemistry and fluo-rophores (FAM, 6-carboxy-fluorescein; HEX, hexacholo-6-carboxy-fluorescein; TET, tetrahydrochloro-6-carboxy-fluorescein) were covalently linked to the 5'-end via iodoacetamide (Molecular Probes, Eugene, OR) or phosphoramidite derivatives (Glen Research, Sterling, VA). After release from the solid support, molecular beacons were purified twice by reverse phase high pressure liquid chromatography using Waters 600E equipped with a Waters 996 Photodiode array detector for simultaneous detection at three different wavelengths. Ion molecular weights of purified molecular beacons were determined by mass spectrometry using a Dynamo Time-Off-Flight instrument.


View this table:
[in this window]
[in a new window]
  		Table 1. Molecular Beacon Sequences

 
To ensure that the selected molecular beacon probe sequences provided maximum sequence discrimination at the PCR-annealing temperature, melt curves were generated for each molecular beacon probe (200 nmol/L in 100 mmol/L KCl, 5 mmol/L MgCl2, 20 mmol/L Tris-HCl, pH 8.0) by measuring fluorescence as a function of temperature in the presence of fourfold excess oligodeoxynucleotide targets specific for all seven HPV types as well as in absence of target

Molecular Beacon-PCRs

PCR conditions for the ß globin gene PCR with the RS40/RS4225 and the HPV-PCR with the CPI/CPIIG primers13 amplification were identical. All PCR reactions were performed in the presence of 1x PCR buffer (15 mmol/L Tris/HCl, pH 8.0, 50 mmol/L KCl), 5 mmol/L MgCl2, 200 µmol/L of deoxynucleotide-triphosphate (dATP, dCTP, dGTP, dTTP), 1 U of Amplitaq Gold DNA polymerase (all from Applied Biosystems, The Netherlands), 250 mmol/L of each adequate primer, 200 nmol/L of each molecular beacon, and 50 µl of sample DNA in a 50-µl final volume. PCR amplification reactions were performed in an ABI Prism 7700 spectrofluorometric thermal cycler (Applied Biosystems, The Netherlands) with the following thermal cycle conditions: 95°C denaturation and enzyme activation for 10 minutes followed by 40 cycles of 95°C denaturation for 15 seconds, 50°C annealing for 50 seconds, and 72°C elongation for 1 minute. Fluorescence spectra were recorded during the annealing phase of each PCR cycle. The Sequence Detection Software (SDS v1.7) of the ABI-Prism 7700 was used to generate the amplification curves.

General HPV-PCR Reporting with SybrGreen

Amplification reactions were performed with 1x SybrGreen PCR Master Mix (Applied Biosystems), 200 nmol/L of CPII/G/CPI primers and 5 µl of sample DNA in a 50-µl final volume. Amplification reactions were performed in an ABI Prism 7700 spectrofluorometric thermal cycler (Applied Biosystems) with PCR conditions identical to the molecular beacon-PCRs. Fluorescence spectra were recorded during the elongation phase of each PCR cycle. The Sequence Detection Software (SDS v1.7) of the ABI-Prism 7700 was used to generate the amplification curves for each reaction.

To differentiate specific amplicons from nonspecific products (typically seen as a smear or primer-dimers on gel), a DNA association curve was generated after each reaction with the ABI Prism 7700 Sequence detection system (Applied Biosystems). After denaturation, the temperature was decreased from 95 to 25°C in 1°C steps with holds for 20 seconds. SybrGreen fluorescence was measured during each step and the first derivative of the fluorescence was plotted as a function of the temperature.

Dilution Series and Multiplexing

For generation of standard lines, dilutions were prepared by mixing SiHa and BSM cells such that between 6 and 6 x 105 copies of HPV-16 were present at a cell concentration of 1000 cells per 5 µl. Single, duplex, and triplex HPV molecular beacon PCR reactions were performed with SiHa, HeLa, CC10A cell lysates and HPV-16, -18, and -45 molecular beacons. All combinations were analyzed (three single, three duplex, one triplex). In addition, dilutions of the HPV-16 (5 x 103, 2.5 x 103, 103, 5 x 102 copies) with a constant background of HPV-18 (105 copies) and HPV-45 (1.2 x 105 copies) were prepared to assess the detection sensitivity and specificity in the presence of multiple HPV types.

Gel Analysis of PCR Reactions

For comparison and estimation of PCR efficacy, all samples amplified in this study were analyzed by standard agarose gel electrophoresis.

Calculations

The threshold cycle number (Ct) was calculated with Sequence Detection System (SDS) software v1.7 (Applied Biosystems) and an automatic setting of the baseline. The baseline value was the average fluorescence value of PCR cycles 3 to 15 plus 10 times its SD. Standard calibration curves were generated by plotting the Ct values against the log of the input copy numbers. The Ct difference ({Delta}Ct = CtHPV - Ctß-globin) was taken as a relative measure of the virus DNA over nuclear genome copy number.


   Results
Top
 Abstract
 Introduction
 Materials and Methods
 Results
Discussion
 References
 
Efficiency of General Primer HPV-PCR

Our previous data showed that the CPI/IIG general primer PCR very efficiently detected HPV-DNA in formalin-fixed paraffin-embedded tissue derived from cervical cancers (Table 2) . This prompted us to design HPV molecular beacons for the 188-bp CPI/IIG-bracketed HPV amplicon for five of the most prevalent HR (HPV-16, -18, -31, -33, and -45) and two of the most prevalent LR HPV types (HPV-6 and HPV-11) (Table 1) . To determine any influence of molecular beacons on the efficiency of the HPV-PCR, the reactions were subjected to agarose gel electrophoresis and analyzed for the presence of the 188-bp product. After breaking the seal at the end of the study, the results were compared to our earlier findings with HPV-PCR using CPI/IIG, MY09-11, GP5–6 primers (Table 2) .


View this table:
[in this window]
[in a new window]
  		Table 2. Comparison of PCR Amplification Efficiency of 53 Patient Samples by Using Different General Primer Sets

 
Characterization of the Type Specificity of the Seven HPV Molecular Beacons

Molecular beacon probes fluoresce on hybrid formation with their target sequences. The maximally discriminating hybridization temperature is dependent on several factors such as GC content, probe (loop), and stem length. To determine the optimal temperature range at which each different HPV molecular beacon probe discriminates among the HPV targets, melt-measurements of the various probe/target combinations were performed. As an example, the melt characteristics of duplexes between the HPV-16 molecular beacon and five target HPV-oligodeoxynucleotide sequences are shown in Figure 1 . The smallest difference between the HPV sequences used in this study is two nucleotides and exists between HPV-16 and HPV-31 (Table 1) . As shown in Figure 1 , at the PCR annealing temperature of 50°C, the duplexes between the HPV-16 molecular beacon and the HPV-31 target fluoresces only slightly more than the HPV-16 molecular beacon without template (dotted line). The other HPV-type targets, because of their greater sequence variation, do not hybridize at all over a large temperature range. The molecular beacons specific for the other six HPV types were analyzed in a similar manner and none of them showed hybrid formation with the nonmatching targets at a temperature >50°C.



View larger version (22K):
[in this window]
[in a new window]
  		Figure 1. Melt-curves of duplexes of the HPV-16 molecular beacon and oligodesoxynucleotide targets of various HPV types. At the PCR annealing temperature (50°C), the HPV-16 molecular beacon clearly discriminates the HPV-16 target from the other types. At lower temperatures, the HPV-31 target oligodesoxynucleotide, which differs at two nucleotide positions from the HPV-16 target, showed a low level of hybridization with the HPV-16 molecular beacon. All other targets have greater sequence variation and do not cross-hybridize, even at lower temperatures.

 
Sensitivity and Specificity of Molecular Beacon PCR on Patient Samples

The four mock tissue samples showed no amplification pattern in the real-time HPV molecular beacon-PCRs. Also after gel electrophoresis no bands were visible.

The 21 tissue samples containing 14 different HPV types not matching with any of the selected HPV molecular beacons, did not report fluorescence amplification in any of the HPV molecular beacon-PCRs, but gave amplification of the 188-bp product as judged by gel electrophoresis (data not shown). Thus no aspecific reporting of molecular beacon probes was detected for the nonmatching types, demonstrating the high degree of specificity of molecular beacon probes.

Of the remaining 32 samples, 25 showed amplification reactions with one and 2 showed amplification with two of the seven HPV molecular beacons. The HPV types as determined by the molecular beacon HPV-PCR were in accordance with the types obtained by the general primer PCR amplification followed by sequencing. In Figure 2 , representative amplification curves for the responding HPV molecular beacons are shown for three HPV-positive patient samples (HPV-16, -18, and -45). All other type-specific HPV molecular beacon-PCRs resulted in curves similar to the one shown for the negative control and these never passed the automatically set threshold.



View larger version (17K):
[in this window]
[in a new window]
  		Figure 2. Comparison of real-time HPV PCR amplification reported by molecular beacons and SybrGreen. A selection of three samples from cervical cancer patients and a negative control (mock tissue section) is shown. HPV molecular beacon PCR (filled symbols): the three patient samples were genotyped as HPV-16 (squares), HPV-45 (triangles), and HPV-18 (circles). The mock tissue was negative with all seven HPV molecular beacons (diamonds). HPV SybrGreen PCR (open symbols): all three patient samples reported positive, whereas the negative control did not. The dashed line represents the threshold.

 
To elucidate the negative results for five previously positive cases a histological revision of the sections was performed, as described by Walboomers and colleagues.7 After histological revision, three cases could be explained as being caused by sampling errors. Indeed, we found similar efficiencies with CPI/IIG primed HPV molecular beacon-PCR as with PCR using CPI/IIG primers alone (see Table 2 ), proving that the presence of molecular beacon probes does not influence the amplification reaction. The remaining two cases were negative by HPV molecular beacon-PCR and no amplicons were present after gel electrophoresis, but they were positive with the ß-globin molecular beacon-PCR. Revising the original genotyping results, the typing proved to be obtained with another general primer set and these cases also then proved not amplifiable with the CPI/CPIIG primer set (see Table 1 and legend, representing a case of HPV-16 and HPV-45). Possibly, the loss of the E1 region because of integration is a cause of these negative PCR reactions.

Multiplexed HPV Molecular Beacon PCR

As a screening method, HPV molecular beacon PCR for seven HPV types suffers from the drawback that an estimated 10 to 20% of total HPV infection cases will not be reported.6,28 Expansion of the number of types in the HPV molecular beacon-PCR would decrease this false-negative rate. In view of the limited availability of sample DNA and for assay convenience, multiplexing of HPV molecular beacon-PCR then becomes desirable. To show the feasibility of multiplexing, duplexed and triplexed HPV molecular beacon-PCRs were performed with molecular beacons for HPV-16, -18, and -45 labeled with FAM, TET, and HEX, respectively. Double (HPV-16 to -18, HPV-16 to -45, HPV-18 to -45) and triple (HPV-16 to HPV-18 to HPV-45) infections were simulated by mixing SiHa, HeLa, and CC10A DNA and treating such simulation samples identical to the patient samples from the archive. As shown in Figure 3 , at approximately equal cellular DNA input corresponding to ~5 x 103, 105, 1.2 x 105 HPV-16, -18, and -45 molecules, respectively, an unequivocal detection was observed in all cases.



View larger version (25K):
[in this window]
[in a new window]
  		Figure 3. Multiplexed HPV molecular beacon PCR. Molecular beacons for HPV-16, -18, and -45, labeled, respectively, with FAM, TET, and HEX, were mixed and used in HPV PCRs with lysates of SiHa, HeLa, and CC10A cells to stimulate single infections (black-filled symbols) and a mixture of all three cells to simulate a triple infection with HPV-16 present at 1% of the input of HPV-18 and -45 (open symbols). A no-template control was included as well. In the single infection simulation, the HPV-16 FAM molecular beacon was the only molecular beacon responding with SiHa cells (black-filled squares). Similarly, the HPV-18 TET molecular beacon only responded with HeLa cells (black-filled triangles) and the HPV-45 HEX molecular beacon only with CC10A (black-filled circles). The beacons not expected to respond in these single infection simulations did not pass the threshold, as did all three HPV molecular beacons in the no-template control (gray-filled symbols). In the triple infection simulation, all three molecular beacons responded (open symbols) and as expected with the low copy number HPV-16 input, the HPV-16 molecular beacon (open squares) passed the threshold some six cycles later than the HPV-18 and -45 molecular beacons (open triangles and open circles, respectively).

 
Also in cases in which large HPV copy number differences were modeled in triple infection, the triplexed HPV molecular beacon PCR performed well: a proportion of 1% of HPV-16 DNA was detectable (2.5 x 103 HPV-16 molecules in the presence of 105 of HPV-18 and 1.2 x 105 of HPV-45 molecules). In triple infection simulations, in which the percentage of one type of HPV (HPV-16) was <0.5%, detection was not possible (result not shown).

General Prescreening Strategy by SybrGreen

Although the multiplexing capacity of molecular beacon PCR is higher than three,29 it is unlikely that it will approach the number of different HPV genotypes. For this reason we looked for a HPV-type-independent, homogeneous detection assay that could function as a general detection method for HPV prescreening. SybrGreen is a DNA intercalating fluorescent dye that quantitatively reports the presence of dsDNA. In combination with the CPI/IIG primer set, SybrGreen fluorescence can therefore be used in a homogeneous assay for quantitative monitoring of HPV-DNA amplification in real-time. SybrGreen will also detect any nonspecifically amplified DNA or primer-dimer artifacts. Nonspecific amplification products have a wide range of sizes, whereas a specifically amplified HPV product is of defined size, as are primer-dimer artifacts. These size differences of DNA fragments will result in differences in temperatures of maximum reassociation rate. Such information can be used to discriminate nonspecific amplification and primer-dimers from HPV-specific amplification.

In Figure 4A the SybrGreen HPV-PCR amplification patterns obtained with SiHa cells diluted in BSM cells (corresponding to 50, 25, 12, and 2 copies of HPV-16 in a total of 1000 cells) are shown, whereas Figure 4B shows the first derivatives of the SybrGreen association curves. For the higher copy number inputs, clear peaks can be discerned at 73°C. Only when the input HPV copy number is as low as 2 per 1000 cells, a clear peak around 73°C cannot be discerned anymore. Also in Figure 4 , the results obtained of the HPV SybrGreen-PCR of the ovarian cancer cases are presented. In 22 out of the 25 cases, no SybrGreen signal was observed after PCR cycle number 40, but three samples (12%) gave a SybrGreen signal over the threshold at high-cycle number (n >= 36). As can be seen in Figure 4B , two of the three differentials of the SybrGreen association curves show peaks quite distinct from the HPV amplicon, whereas the third resembles the one of the lowest HPV input.



View larger version (22K):
[in this window]
[in a new window]
  		Figure 4. HPV SybrGreen PCR and differentials of association curves of PCR products. A: Real-time SybrGreen amplification patterns of four SiHa cell line dilutions corresponding to an input of 50, 25, 12, and 2 HPV genomes (A) and a selection of 6 of the 25 ovarian cancer samples. The four SiHa cell line dilutions and 3 of the 25 ovarian cancer cases passed the threshold (B). For reasons of clarity, only 5 of the 22 other ovarian cancer samples not passing the threshold are depicted (C). A (mock tissue) negative control (D) is also shown. B: The differentials of the association curves of the samples in A (identical labeling patterns). With two exceptions, the low-copy number HPV input sample and one of the three ovarian cancer samples that surpassed the SybrGreen threshold, the curves can be placed in three clusters: one with a peak around 80°C (representing fragmented high-molecular weight DNA that smears higher up on a gel), one with a peak around 73°C (representing HPV amplicons that give a distinct band on gel), and one with a peak around 63°C (representing fragmented low-molecular weight DNA and/or primer dimers that run low in the gel).

 
In addition, the 22 SybrGreen-negative ovarian cancer samples were analyzed for the shape of the differential of the association curves. Two clusters could be discerned. One with curves having peaks around 80°C and one around 63°C. The 80°C peak proved to represent samples with fragmented, relatively high molecular weight DNA and relatively high cell content. The 63°C peak represents samples with fragmented, degraded, relatively low molecular weight DNA as well as primer-dimers artifacts. By rating differential association curves of positive HPV SybrGreen PCRs that have a distinct peak either at 80 or 63°C as HPV-negative, 1 of the 25 samples is to be considered positive in the SybrGreen analysis.

By gel electrophoresis, none of the ovarian cancer samples was positive. Hence, the false-positive rate of the HPV SybrGreen PCR assay is 12%. By combining it with the association analysis, the false-positive rate of SybrGreen analysis can be reduced to 4%.

Feasibility of the Two-Step HPV Detection and Genotyping Strategy

To assess the feasibility of the HPV detection and genotyping strategy, 19 additional samples, derived from eight patients were selected. ß-globin molecular beacon-PCRs, were performed and all samples showed a positive amplification pattern, confirming the suitability of the DNA for PCR amplification.

To determine the presence of HPV in samples, SybrGreen HPV-PCR was performed and 7 out of the 19 samples were positive. The seven positive samples also had peaks characteristic for HPV amplicons in the association analysis, including one histologically tumor-free lymph node (data not shown). Ten of the 12 samples negative in the SybrGreen analysis were from tumor-free operation edges or lymph nodes. The remaining two were cases of cervical cancer in which further analysis showed that these samples would not PCR amplify with the CPI/IIG primer set. Using other primer sets and sequencing, one of these cases was found to be HPV-18, whereas the other could not be typed.

In the HPV molecular beacon PCR, five of the seven SybrGreen-positive samples were positive: one (laryngeal papilloma) was genotyped as HPV-11 and four others (one adenocarcinoma and its tumor-free lymph node, two squamous cell carcinomas) as HPV-16. The three HPV SybrGreen-PCR-positive, but HPV molecular beacon-PCR-negative samples proved to be of a HPV type not included in the current HPV molecular beacon set (HPV-58 and HPV-68) based on data obtained by sequencing.

Quantitating Viral over Nuclear Genome Copy Number

Real-time monitoring of PCR amplification allows a reliable determination of the input copy number using the Ct value.30 Thus viral and nuclear genome copy numbers can be determined, respectively, with the SybrGreen or HPV molecular beacon PCR and the ß-globin molecular beacon PCR. As shown in Figure 2, a good correlation was observed between the HPV SybrGreen and molecular beacon PCR.

The difference in Ct values ({Delta}Ct) of the HPV and ß-globin PCRs provides a measure of the viral DNA load per cell. Prerequisite for this {Delta}Ct approach is that the efficiencies of the two PCR reactions are approximately equal over a wide input range. As can be seen in Ct versus SiHa cell input plots of Figure 5 , the lines for the ß-globin and HPV SybrGreen-PCR run together throughout 4 decades. Thus these PCRs, which run under identical cycling conditions, show equal efficiencies, implying that the difference in Ct values ({Delta}Ct) may be used as a measure of virus load per cell. From these data, it is obvious that the present HPV detection strategy not only allows qualitative determination, but also quantitation down to 50 cell equivalents with 1 to 2 copies of HPV per cell. For all cervical and ovarian samples analyzed, Ct value for the ß-globin PCR ranged between 19.5 and 36.2. For the cervical cancer cases, the Ct value of the HPV-PCRs ranged between 18.2 and 35.8. The highest viral copy number was obtained with a HPV-6-positive case with Ct values for HPV and ß-globin 18.2 and 31.5, respectively, indicating 104 HPV copy number excess over the ß-globin gene. This finding was in line with the histological findings that showed a small biopsy sample of a CIN lesion.



View larger version (23K):
[in this window]
[in a new window]
  		Figure 5. Ct values of the ß-globin molecular beacon PCR (closed circles) and the HPV-CPI/IIG-SybrGreen PCR (open squares) as a function of the amount of input SiHa cell DNA. SiHa carries one to two copies of HPV-16. Bars indicate standard errors of three independent experiments, the range of standard errors: 0.1 to 0.76 cycles. With the {Delta}Ct approach accurate quantification of viral over nuclear genome copy number is possible even with 20 cell equivalents.

 

   Discussion
Top
 Abstract
 Introduction
 Materials and Methods
 Results
 Discussion
References
 
Currently, the diagnosis of HPV infection is based on detection of the viral genome. The large number of HPV types and their subdivision in high-risk and low-risk types requires a detection methodology that is broad spectrum as well as type-specific. We have developed a novel strategy for both general and type-specific detection of HPV using CPI/IIG primers and real-time PCR with SybrGreen and molecular beacon reporting, which allows quantitation of the viral load in cervical cancer samples derived from formaldehyde-fixed tissues.

Molecular beacons were designed for typing the seven prevalent anogenital HPV types. On the basis of results obtained with 21 different HPV types represented in 53 formaldehyde-fixed samples, it can be concluded that the HPV molecular beacon probes are highly type-specific and well suited for application in a multiplexed homogeneous PCR assay for rapid genotyping of HPV sequences.

Because of the large sequence diversity of HPV types on one hand and the enhanced sequence variant discrimination of molecular beacons on the other,31 a single molecular beacon probe that detects all types is not feasible. To detect the presence of HPV in a type-independent manner and in a homogeneous format, we have evaluated the use of SybrGreen as a general reporter of HPV-PCR amplification. The false-positive rate of the HPV SybrGreen-PCR proved to be 12% (n = 25) when using signal-over-threshold as a criterion for positivity. However, the false-positive rate could be reduced to 4% by performing a PCR product-association analysis.

Techniques that use processing after PCR for detecting and genotyping HPV are not only tedious, but also have limited capacity for quantitating the (average) viral genome load per cell. Quantification of DNA or RNA copy numbers, based on an external calibration curve constructed from cloned or cell line DNA has a limited value for samples with highly variable quality such as formaldehyde-fixed tissues. This variability may cause 100 to 1000 times underestimation of the actual copy number.32 The fact that the ß-globin molecular beacon and HPV SybrGreen-PCR have identical efficiencies (Figure 5) enables one to use the {Delta}Ct value of these PCRs for an HPV-type-independent quantitation of the average HPV copy number per ß-globin gene that is also independent of the DNA quality. Tumor samples are often contaminated by normal cells, the viral load per cell determinations are therefore averages, implying that they must be interpreted in the histological/cytological context of the sections obtained before and after the cut for the DNA isolation.

The ß-globin molecular beacon-PCR for assessment of DNA quality and nuclear genome quantity, the SybrGreen HPV-PCR for general HPV detection and quantitation and the HPV molecular beacon-PCR for genotyping the most prevalent HPV types allow an attractive HPV-screening and -genotyping strategy, see Figure 6 . We showed the potential of this strategy on eight new cases (19 samples). Intriguingly, one of the samples was a histologically tumor-free lymph node that showed positive amplification in the SybrGreen detection and that was subsequently typed HPV-16 by the molecular beacon-PCR. This finding indicates the sensitivity of our method, and shows its potential for detecting minimal residual disease.33



View larger version (29K):
[in this window]
[in a new window]
  		Figure 6. Strategy for general and type-specific detection and quantification of HPV.

 
Many of the advantages of the proposed screening/genotyping strategy stems from the fact that it can be performed under homogeneous assay conditions in 96-well format. The real-time data collection avoids sample handling after PCR that reduces the risk of cross-contamination. Furthermore, the fluorescence detection permits multiplexing, which reduces the number of reactions to be performed and consequently the material demand, which in cervical smear analysis is of importance. Recent reports showed that in HPV infection not only the HPV type but also that the viral load and its long-term persistence are risk factors for progression to cervical cancer.34,35

Other studies describe detection of HPV in patient samples under homogenous (in casu closed tube) assay conditions. The strategy proposed here encompasses many of the favorable features of homogeneous detection assays described in these studies. Strauss and colleagues36 used general primer-based amplification reported by SybrGreen in real-time for a type-independent prescreening for the presence of HPV in samples, whereas Jordens and colleagues37 used molecular beacons coupled to a general primer, comparable to the Sunrise primer system38 to accomplish the same. In our strategy a similar prescreening is performed by SybrGreen reporting, whereas molecular beacons are used for the genotyping of the most prevalent high-risk and low-risk HPV types. It reveals the genotype information for the majority of the samples. Swan and colleagues39 used a hybridization probe-based assay with HPV-type-specific primer PCR and detection by a TaqMan probe in an end-point readout format. Although highly specific this approach has limited capacity for detection of multiple types and has also limiting quantitation potential. Joseffson and colleagues32 used the same TaqMan-probing principle in combination with a general primer pair and real-time read out, but did not incorporate general reporting for the remaining HPV types. The sensitivities of our method and the TaqMan approach are similar, but molecular beacon probes have a higher degree of specificity compared to TaqMan probes.40 Indeed, using the set of seven different probes on 21 types of HPV target we have not found any nonspecific reporting of nonmatching types nor in the included negative controls, contrary to the method described by Josefsson and colleagues32 in which the rate of false-positive reporting was found to be 10%.

The real-time PCR assay described here is a fast and simple method to quantitatively detect and type HPV DNA in cervical cancer. It can easily be extended to other HPV types and to tissue or cell samples.


   Acknowledgements
 
We thank Dr. Fred Kramer and Salvatore Marras for advice on molecular beacons, and Dario Largana and Michel Dechamps for technical assistance in the synthesis of the probes used in this study.


   Footnotes
 
Address reprint requests to Dr. Anton K. Raap, Laboratory for Cytochemistry and Cytometry, Department of Molecular Cell Biology, Leiden University Medical Center, Wassenaarseweg 72, 2333AL Leiden, The Netherlands. E-mail: a.k.raap{at}lumc.nl

Current address of E. S.: Department of Pathology, University Hospital Groningen, Groningen, The Netherlands.

Accepted for publication July 18, 2001.


   References
Top
 Abstract
 Introduction
 Materials and Methods
 Results
 Discussion
 References
 

  1. zur Hausen H: Papillomaviruses causing cancer: evasion from host-cell control in early events in carcinogenesis. J Natl Cancer Inst 2000, 92:690-698[Abstract/Free Full Text]
  2. Pfister H, ter Schegget J: Role of HPV in cutaneous premalignant and malignant tumors. Clin Dermatol 1997, 15:335-347[Medline]
  3. Ryan DP, Compton CC, Mayer RJ: Carcinoma of the anal canal. N Engl J Med 2000, 342:792-800[Free Full Text]
  4. Gillison ML, Koch WM, Capone RB, Spafford M, Westra WH, Wu L, Zahurak ML, Daniel RW, Viglione M, Symer DE, Shah KV, Sidransky D: Evidence for a causal association between human papillomavirus and a subset of head and neck cancers. J Natl Cancer Inst 2000, 92:709-720[Abstract/Free Full Text]
  5. Frisch M, Biggar RJ: Aetiological parallel between tonsillar and anogenital squamous-cell carcinomas. Lancet 1999, 354:1442-1443[Medline]
  6. Bosch FX, Manos MM, Munoz N, Sherman M, Jansen AM, Peto J, Schiffman MH, Moreno V, Kurman R, Shah KV: Prevalence of human papillomavirus in cervical cancer: a worldwide perspective. International biological study on cervical cancer (IBSCC) Study Group. J Natl Cancer Inst 1995, 87:796–802
  7. Walboomers JM, Jacobs MV, Manos MM, Bosch FX, Kummer JA, Shah KV, Snijders PJ, Peto J, Meijer CJ, Munoz N: Human papillomavirus is a necessary cause of invasive cervical cancer worldwide. J Pathol 1999, 189:12-19[Medline]
  8. zur Hausen H: Papillomavirus infections—a major cause of human cancers. Biochim Biophys Acta 1996, 1288:F55-F78[Medline]
  9. Nobbenhuis MA, Walboomers JM, Helmerhorst TJ, Rozendaal L, Remmink AJ, Risse EK, van der Linden HC, Voorhorst FJ, Kenemans P, Meijer CJ: Relation of human papillomavirus status to cervical lesions and consequences for cervical-cancer screening: a prospective study. Lancet 1999, 354:20-25[Medline]
  10. Lorincz AT: Molecular methods for the detection of human papillomavirus infection. Obstet Gynecol Clin North Am 1996, 23:707-730[Medline]
  11. Peyton CL, Schiffman M, Lorincz AT, Hunt WC, Mielzynska I, Bratti C, Eaton S, Hildesheim A, Morera LA, Rodriguez AC, Herrero R, Sherman ME, Wheeler CM: Comparison of PCR- and hybrid capture-based human papillomavirus detection systems using multiple cervical specimen collection strategies. J Clin Microbiol 1998, 36:3248-3254[Abstract/Free Full Text]
  12. Baay MF, Quint WG, Koudstaal J, Hollema H, Duk JM, Burger MP, Stolz E, Herbrink P: Comprehensive study of several general and type-specific primer pairs for detection of human papillomavirus DNA by PCR in paraffin-embedded cervical carcinomas. J Clin Microbiol 1996, 34:745-747[Abstract]
  13. Tieben LM, ter Schegget J, Minnaar RP, Bouwes Bavinck JN, Berkhout RJ, Vermeer BJ, Jebbink MF, Smits HL: Detection of cutaneous and genital HPV types in clinical samples by PCR using consensus primers. J Virol Methods 1993, 42:265-279[Medline]
  14. Manos MM, Ting Y, Wright DK, Lewis AJ, Broker TR, Wolinsky SM: The use of polymerase chain reaction amplification for the detection of genital human papillomaviruses. Cancer Cells 1989, 7:209-214
  15. Roda Husman AM, Walboomers JM, van den Brule AJ, Meijer CJ, Snijders PJ: The use of general primers GP5 and GP6 elongated at their 3' ends with adjacent highly conserved sequences improves human papillomavirus detection by PCR. J Gen Virol 1995, 76:1057-1062[Abstract/Free Full Text]
  16. Kleter B, van Doorn LJ, ter Schegget J, Schrauwen L, van Krimpen K, Burger M, ter Harmsel B, Quint W: Novel short-fragment PCR assay for highly sensitive broad-spectrum detection of anogenital human papillomaviruses. Am J Pathol 1998, 153:1731-1739[Abstract/Free Full Text]
  17. Gravitt PE, Peyton CL, Apple RJ, Wheeler CM: Genotyping of 27 human papillomavirus types by using L1 consensus PCR products by a single-hybridization, reverse line blot detection method. J Clin Microbiol 1998, 36:3020-3027[Abstract/Free Full Text]
  18. Kleter B, van Doom LJ, Schrauwen L, Molijn A, Sastrowijoto S, ter Schegget J, Lindeman J, ter Harmsel B, Burger M, Quint W: Development and clinical evaluation of a highly sensitive PCR-reverse hybridization line probe assay for detection and identification of anogenital human papillomavirus 1. J Clin Microbiol 1999, 37:2508-2517[Abstract/Free Full Text]
  19. Coutlee F, Mayrand MH, Provencher D, Franco E: The future of HPV testing in clinical laboratories and applied virology research. Clin Diagn Virol 1997, 8:123-141[Medline]
  20. Livak KJ, Marmaro J, Todd JA: Towards fully automated genome-wide polymorphism screening. Nat Genet 1995, 9:341-342[Medline]
  21. Tyagi S, Kramer FR: Molecular beacons: probes that fluoresce upon hybridization. Nature Biotechnol 1996, 14:303-308[Medline]
  22. Koopman LA, Szuhai K, van Eendenburg JD, Bezrookove V, Kenter GG, Schuuring E, Tanke H, Fleuren GJ: Recurrent integration of human papillomaviruses 16, 45, and 67 near translocation breakpoints in new cervical cancer cell lines. Cancer Res 1999, 59:5615-5624[Abstract/Free Full Text]
  23. Krul EJ, Kersemaekers AM, Zomerdijk-Nooyen YA, Cornelisse CJ, Peters LA, Fleuren GJ: Different profiles of allelic losses in cervical carcinoma cases in Surinam and The Netherlands. Cancer 1999, 86:997-1004[Medline]
  24. Anttila M, Syrjanen S, Ji H, Saarikoski S, Syrjanen K: Failure to demonstrate human papillomavirus DNA in epithelial ovarian cancer by general primer PCR. Gynecol Oncol 1999, 72:337-341[Medline]
  25. Greer CE, Peterson SL, Kiviat NB, Manos MM: PCR amplification from paraffin-embedded tissues. Effects of fixative and fixation time. Am J Clin Pathol 1991, 95:117-124[Medline]
  26. SantaLucia J, Jr: A unified view of polymer, dumbbell, and oligonucleotide DNA nearest-neighbor thermodynamics. Proc Natl Acad Sci USA 1998, 95:1460-1465[Abstract/Free Full Text]
  27. Szuhai K, Ouweland JM, Dirks RW, Lemaitre M, Truffert JC, Janssen GM, Tanke HJ, Holme E, Maassen JA, Raap AK: Simultaneous A8344G heteroplasmy and mitochondrial DNA copy number quantification in myoclonus epilepsy and ragged-red fibers (MERRF) syndrome by a multiplex molecular beacon based real-time fluorescence PCR. Nucleic Acids Res 2001, 29:E13
  28. Pirog EC, Kleter B, Olgac S, Bobkiewicz P, Lindeman J, Quint WG, Richart RM, Isacson C: Prevalence of human papillomavirus DNA in different histological subtypes of cervical adenocarcinoma. Am J Pathol 2000, 157:1055-1062[Abstract/Free Full Text]
  29. Tyagi S, Marras SA, Kramer FR: Wavelength-shifting molecular beacons. Nature Biotechnol 2000, 18:1191-1196[Medline]
  30. Higuchi R, Fockler C, Dollinger G, Watson R: Kinetic PCR analysis: real-time monitoring of DNA amplification reactions. Biotechnology 1993, 11:1026-1030[Medline]
  31. Tyagi S, Bratu DP, Kramer FR: Multicolor molecular beacons for allele discrimination. Nature Biotechnol 1998, 16:49-53[Medline]
  32. Josefsson A, Livak K, Gyllensten U: Detection and quantitation of human papillomavirus by using the fluorescent 5' exonuclease assay. J Clin Microbiol 1999, 37:490-496[Abstract/Free Full Text]
  33. Kobayashi Y, Yoshinouchi M, Tianqi G, Nakamura K, Hongo A, Kamimura S, Mizutani Y, Kodama J, Miyagi Y, Kudo T: Presence of human papilloma virus DNA in pelvic lymph nodes can predict unexpected recurrence of cervical cancer in patients with histologically negative lymph nodes. Clin Cancer Res 1998, 4:979-983[Abstract]
  34. Ylitalo N, Sorensen P, Josefsson AM, Magnusson PK, Andersen PK, Ponten J, Adami HO, Gyllensten UB, Melbye M: Consistent high viral load of human papillomavirus 16 and risk of cervical carcinoma in situ: a nested case-control study. Lancet 2000, 355:2194-2198[Medline]
  35. Josefsson AM, Magnusson PK, Ylitalo N, Sorensen P, Qwarforth-Tubbin P, Andersen PK, Melbye M, Adami HO, Gyllensten UB: Viral load of human papilloma virus 16 as a determinant for development of cervical carcinoma in situ: a nested case-control study. Lancet 2000, 355:2189-2193[Medline]
  36. Strauss S, Jordens JZ, Desselberger U, Gray JJ: Single-tube real-time nested polymerase chain reaction for detecting human papillomavirus DNA. Diagn Mol Pathol 2000, 9:151-157[Medline]
  37. Jordens JZ, Lanham S, Pickett MA, Amarasekara S, Abeywickrema I, Watt PJ: Amplification with molecular beacon primers and reverse line blotting for the detection and typing of human papillomaviruses. J Virol Methods 2000, 89:29-37[Medline]
  38. Nazarenko IA, Bhatnagar SK, Hohman RJ: A closed tube format for amplification and detection of DNA based on energy transfer. Nucleic Acids Res 1997, 25:2516-2521[Abstract/Free Full Text]
  39. Swan DC, Tucker RA, Holloway BP, Icenogle JP: A sensitive, type-specific, fluorogenic probe assay for detection of human papillomavirus DNA. J Clin Microbiol 1997, 35:886-891[Abstract]
  40. Tapp I, Malmberg L, Rennel E, Wik M, Syvanen AC: Homogeneous scoring of single-nucleotide polymorphisms: comparison of the 5'-nuclease TaqMan assay and molecular beacon probes. Biotechniques 2000, 28:732-738[Medline]



This article has been cited by other articles:


Home page
 J. Clin. Microbiol.Home page
K. Biedermann, N. Dandachi, M. Trattner, G. Vogl, H. Doppelmayr, E. More, A. Staudach, O. Dietze, and C. Hauser-Kronberger
Comparison of Real-Time PCR Signal-Amplified In Situ Hybridization and Conventional PCR for Detection and Quantification of Human Papillomavirus in Archival Cervical Cancer Tissue
J. Clin. Microbiol., August 1, 2004; 42(8): 3758 - 3765.
[Abstract] [Full Text] [PDF]

Home page
 J Med MicrobiolHome page
N. Speich, C. Schmitt, R. Bollmann, and M. Bollmann
Human papillomavirus (HPV) study of 2916 cytological samples by PCR and DNA sequencing: genotype spectrum of patients from the west German area
J. Med. Microbiol., February 1, 2004; 53(2): 125 - 128.
[Abstract] [Full Text] [PDF]

Home page
 J. Immunol.Home page
R. I. Brezinschek, N. Oppenheimer-Marks, and P. E. Lipsky
Activated T Cells Acquire Endothelial Cell Surface Determinants During Transendothelial Migration
J. Immunol., February 1, 1999; 162(3): 1677 - 1684.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Purchase Article
Right arrow View Shopping Cart
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Szuhai, K.
Right arrow Articles by Raap, A. K.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Szuhai, K.
Right arrow Articles by Raap, A. K.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS