New Results

The limitations of in vitro experimentation in understanding biofilms and chronic infection

Aled E. L. Roberts, Kasper N. Kragh, Thomas Bjarnsholt, Stephen P. Diggle
doi: https://doi.org/10.1101/032987

Abstract

We have become increasingly aware that during infection, pathogenic bacteria often grow in multicellular biofilms which are often highly resistant to antibacterial strategies. In order to understand how biofilms form and contribute to infection, in vitro biofilm systems such as microtitre plate assays and flow cells, have been heavily used by many research groups around the world. Whilst these methods have greatly increased our understanding of the biology of biofilms, it is becoming increasingly apparent that many of our in vitro methods do not accurately represent in vivo conditions. Here we present a systematic review of the most widely used in vitro biofilm systems, and we discuss why they are not always representative of the in vivo biofilms found in chronic infections. We present examples of methods that will help us to bridge the gap between in vitro and in vivo biofilm work, so that our bench-side data can ultimately be used to improve bedside treatment.

Introduction

Bacteria were once thought to exist as single, free-floating planktonic cells that are community independent. John William (Bill) Costerton changed this perception in the late 1970s when he observed surface associated microbial aggregates enclosed within a matrix of extracellular material, a phenomena he later termed ‘biofilm’ [1,2]. Today, the biofilm phenotype has been identified in up to 80% of all non-acute infections, including foreign-body related, otitis media, orthopaedic, catheter, chronic wounds and lung-related infections [3–7]. The interchange between planktonic and biofilm phenotypes, is believed, but not proven, to commonly manifest clinically as acute and chronic infections respectively.

Acute infections tend to be fast spreading with a rapid onset. They are often controlled by the host immune response, and excessive intervention is not always required. However, if the host defences fail and therapeutic intervention is required, acute infections can usually be cleared within days [8]. Conversely, chronic infections are where there is a delay in the healing process (an inability of the injured site to restore anatomical and functional integrity), consistent with the severity of the injury [9]. The presence of biofilms and their innate ability to tolerate antibiotics up to 1000 times greater than planktonic cells, is thought to delay wound restoration [10–13]. Cells assuming the biofilm phenotype are commonly observed in patients with various underlying conditions, which can be system wide in the case of immunodeficiency and diabetes, or more focused in the case of venous leg ulcers and cystic fibrosis (CF). In the case of CF, chronic biofilm infections have been known to persist in the airways for over 30 years. Therefore, chronic infections are an ever increasing problem due to their recalcitrance towards extensive antibiotic treatment regimes and persistence under sustained attack from the host’s innate and adaptive immune response systems [10,14].

The treatment and management of patients suffering from chronic infections represents a significant monetary and labour intensive burden to healthcare providers. Recently, the direct costs of chronic infections, such as those affecting the dermis, were estimated to be in excess of $18 billion, affecting 2 million residents in the United States alone, and resulting in 200,000 deaths annually [15]. This data relates to one country, and a single infected organ. If this is representative of other countries, and other conditions, then chronic infections represent a huge worldwide problem. A recent report on antimicrobial resistance (AMR), has stated that we can expect up to 10 million extra deaths annually worldwide by 2050 due to AMR [16]. The problem of chronic infection is only going to aid the rise of AMR.

Much of our current knowledge about chronic infection comes from studying bacteria growing in test tubes. Costerton encouraged laboratories worldwide to deviate from studying planktonic cultures, and instead, focus on understanding surface associated biofilms. This has become increasingly more relevant as we battle with the issues posed by AMR. We are becoming increasingly aware of significant differences that exist between in vitro biofilms grown in the laboratory, and in vivo biofilms found during actual infection. This raises the question as to whether the experiments that we currently perform in the laboratory are useful for understanding how bacterial biofilms form and contribute to AMR during infection.

To understand the biology of infection better, we need another paradigm shift, a new wave of methods and experiments that better represent clinical conditions (of which biofilm formation is only one aspect). The use of some methods can potentially hamper our understanding of various aspects of infection, as they do not always accurately represent what we observe clinically. This review summarises what we know about bacteria during infection, and how our current in vitro methods fail to represent such factors. In the following sections we discuss biofilms and polymicrobial interactions, particularly in the context of Pseudomonas aeruginosa as one of the most common opportunistic pathogens that causes chronic infection. We explore the differences between in vitro and in vivo observations, and discuss how to better bridge the gap between the two, increasing experimental accuracy so that our bench-side data can be used to improve bedside treatment.

The role of biofilms during chronic infection

Chronic infections persist despite apparently adequate antibiotic therapy, and in the face of the host’s innate and adaptive defence mechanisms. Chronic infections are characterised by persistent and progressive pathology, mainly due to the inflammatory response surrounding in vivo biofilms [17]. This biofilm lifestyle appears to impair the host’s ability to combat the infectious agent. The innate immune response in the form of recruitment of neutrophils and their inability to break through the biofilms defence has been specifically examined [18–21]. Polymorphonuclear leukocytes (PMNs) are recruited in large numbers to the infection site, and during acute infection, are able to phagocytize and remove most of the infectious agent [3,18]. When PMNs fail to eradicate an infection, it is most likely because a biofilm has been established. In the biofilm state, the enclosing matrix of extracellular substances is capable of protecting underlying cells from the immune system, such as PMN phagocytosis [22,23]. In addition to this, biofilms are capable of suppressing the antimicrobial action of PMNs through the production of various virulence factors [19,24]. P. aeruginosa growing in biofilms has been shown to actively kill PMNs through the secretion of rhamnolipids which reduce the host’s ability to clear infection [25].

In contrast to acute infections, which are usually treatable by traditional antibiotics, biofilms are known to tolerate antibiotic concentrations up to 1000 times higher than the minimal inhibitory concentrations (MIC). It is important to note that there are differences between antibiotic resistance and tolerance, with many studies reporting the former when meaning the latter. Antibiotic resistance is where there is an increase in the MIC through mechanistic intervention. Conversely, little is known about antibiotic tolerance. There is often little change in the MIC of individual cells, however growth in populations can lead to an increasing tolerance to antibiotics. The cause of such increased tolerances seen in biofilms has been investigated in a number of different studies, focusing on the extracellular matrix, the involvement of quorum sensing (QS), and the physiological factors observed within the biofilm [19,26–29]. The extracellular matrix itself has been shown to obstruct the diffusion of some antimicrobial agents through both chemical and physiological means [27,30,31]. The matrix however is not selective against antimicrobial agents, and therefore the diffusion of many substances, such as oxygen, metabolites and waste products, can also be altered. Reduced diffusion in combination with high cell densities results in steep chemical gradients from the outer surface towards the central core of an in vitro biofilm [26,32]. This causes a very heterogenic growth pattern throughout in vitro biofilms, which in turn influences regional tolerance towards different types of antibiotic treatments [33,34]. Tobramycin tolerance in P. aeruginosa has also been shown to be influenced by QS-systems [19]. These studies have predominantly used in vitro systems in an attempt to explain the increased persistence and tolerance of biofilm cells towards varying antibiotic treatment regimens, and the failures of the host immune system to eradicate infection. However, such studies are very difficult to transfer directly to in vivo settings.

Another observation of chronic infection, is the presence of multiple species within an infection site [35,36]. The milieu of species within a defined space may result in cooperation and/or conflict with other community members [37–39]. This creates a complex environment where species align along different nutrient gradients, be those host derived or the result of species interactions, as seen between P. aeruginosa and Staphylococcus aureus in chronic wounds [40–42]. The complexity of interactions between species may be enhanced through the production of diverse QS signals within the infection site, which have inter- and intra-species effects, although this is yet to be demonstrated in vivo [43–45]. One thing of note, is the lack of evidence supporting multi-species biofilms during infections, whereby species grow concomitantly within the same biofilm, however the presence of multiple species within an infection site is well documented.

In vitro investigation of biofilms

During the last three decades, biofilms of pathogenic species have been extensively studied by a wide range of research groups, each with differing objectives, but all with the same overall aim – to expand our knowledge of biofilms to better understand infection. Using continuous flow-cell conditions and Confocal Laser Scanning Microscopy (CLSM), we are closer to understanding the processes involved in the initial attachment of cells to surfaces in vitro [46–48]. Combining molecular techniques with CLSM to construct and visualise knock-out strains of bacteria, has shown the contribution of motility and QS to biofilm development [30,33,46,47,49].

The development of low cost, high throughput biofilm screening methods in microtitre-plates, have made it possible to identify genes essential for surface-attached biomass production in liquid media [50,51]. This system has been heavily used to identify potential anti-biofilm agents, by measuring the reduction in surface attached biomass on the sides of wells after treatment with potential therapeutic agents [51–54]. In addition to this, The Center for Disease Control (CDC) approved biofilm reactor [55–57], and drip flow reactors, have proved excellent for assessing biofilm formation on biological and non-biological materials [58–60]. These flow systems and microtitre assays are the workhorses of in vitro biofilm research, and have generated a phenomenal amount of data that has greatly expanded our knowledge about how bacteria attach and differentiate into mature biofilms in vitro.

The commonality between these methods is the growth of biofilms on abiotic surfaces, that are submerged in media, and exposed to fluid dynamics of varying degrees. Experiments in these systems are able to produce biofilms of high cell density with a topography that can reach up to several hundred μm thick. Under continuous flow cell conditions, the emergence of mushroom structures and water channels occur after only a couple of day’s growth (Figure 1) [46,47]. A key question is whether we are able to transfer our in vitro knowledge from the laboratory bench to the patient bedside, and the last decade has provided us with refined techniques that allow us to gain insight into what is actually occurring within certain types of infection.

Figure 1.
Figure 1.

Confocal laser scanning micrographs of 2-day-old (A), 3-day-old (B) and 4-day-old (C) biofilms formed by a P. aeruginosa wild-type strain in a continuous flow cell system. The central images show top-down views, and the flanking images show vertical optical sections. The bars represent 10 μm (A) and 20 μm (B and C). Bjarnsholt, unpublished.

With advanced microscopy techniques, we are now able to see how bacterial cells organise themselves in chronic infections in ex vivo samples. As with the majority of biofilm research, there has been considerable focus on infection in patients with CF. In ex vivo lung tissue infected with the major CF pathogen P. aeruginosa, much of the bacterial biomass is found as a biofilm within the bronchial lumen. Interestingly, these biofilms have not been found attached to the epithelial surface, but are found as non-attached biofilm aggregates that are embedded in the highly inflamed mucus [3,21] (Figure 2). Lying completely surrounded by immune cells, mainly PMNs, these aggregates range in size from 4 μm – 100 μm in diameter [3,20]. The same mode of non-attached aggregated biofilm growth have been observed in other types of infection, such as those found in chronic wounds, otitis media, and soft tissue fillers [4,5,61].

Figure 2.
Figure 2.

Micrograph of P. aeruginosa infected lung tissue from a patient with cystic fibrosis. Light and fluorescence microscopy images (170× magnification) of PAS hematoxylin-stained (A, B) and PNA FISH-stained (C, D) sections containing luminal and mucosal accumulations of inflammatory cells. The P. aeruginosa-positive areas are seen as well-defined lobulated clarifications surrounded by inflammatory cells. Red arrows indicate PMNs and green arrows indicate P. aeruginosa biofilm aggregates [21].

A glaring discrepancy between ex vivo observations and the in vitro biofilm, is the absence of a surface. In all of the infections mentioned above, attachment to a surface or the epithelia is rarely observed. However, the majority of primary in vitro models for biofilm infection have surface attachment as a crucial component. Recent studies have shown how P. aeruginosa non-attached biofilm aggregates form in liquid batch cultures, and that they have the same characteristics as surface attached biofilms when it comes to antibiotic tolerance and resilience towards PMNs [62,63]. The absence of such surfaces in surface wounds means the bacterial cells assume a biofilm lifestyle within a self-contained aggregate, devoid of surface attachment [64,65]. There have been a number of in vitro studies that have observed steep chemical gradients (notably oxygen) throughout biofilms, which leads to a heterogenic growth pattern [26,32]. The complex environment inside inflamed mucus seems to govern the growth pattern in biofilms in a different way. When the growth rate of aggregated bacteria within mucus is measured, a heterogenic growth pattern is observed, due to large regional variations between aggregates which can be linked to the local concentration of PMNs [21]. Measurements of expectorated sputum from CF patients with a P. aeruginosa infection, demonstrates a micro-aerophilic environment as a result of oxygen consumption by PMNs [66]. These findings point towards immune cells playing a central role in regulating the growth of bacterial cells as they grow in aggregates, at least during infection in the CF lung. Crucially, immune cells are absent in almost all in vitro systems.

Some prominent biofilm infections do involve the “characteristic” attachment and growth of biofilm cells on a surface. Chronic foreign body infections of implants, stents and catheters, result in bacterial cells colonizing abiotic surfaces, leading to biofilm formation [67–69]. Catheters have been shown to constitute an environment whereby bacterial infections are able to develop into very thick biofilms [67,68,70]. Therefore, catheter associated biofilms are an example of a biofilm that most resembles the flow of media over a surface, and are most similar to CDC reactors, drip flow and flow cell systems. There are still some striking differences. For example, the type of structures observed and the level of organization seen in flow cells, are not present in catheter related biofilms, and mushrooms structures are almost never observed. Other differences include the liquid flowing over the surface. During in vitro biofilm growth, well-defined minimal medias are most often used [49,56,57,71,72]. In catheters, the liquids flowing over the surface could be urine, blood or other bodily fluids, which constitutes a very different growth environment for bacteria. In many other biofilm infections, there is a distinctive lack of flow within the infection site [20]. However, the lack of a flow does not restrict the bacteria from creating a biofilm. For example, P. aeruginosa is capable of biofilm formation deep within the dehydrated mucus of the CF lung, or in the inflamed pus of chronic wounds [73,74].

The pros and cons of each of these methods are listed in Table 1. Many of these methods produce clear and well defined results, due in part to our ability to control experimental parameters with a high degree of stringency, whilst concomitantly allowing single variables to change. This allows us to study the effects of single elements on various aspects of biofilm growth. This reductionist approach has yielded great information about complex cell matters such as metabolism, resistance mechanisms, and signalling pathways. However, such a simplistic approach is not always possible during in vivo methodologies, due to natural variations between living organisms.

View this table:
Table 1.

Comparison of different model systems for studying biofilms in vitro.

In vivo investigation of biofilms

More recently, the advent of in vivo methods have increased our understanding of biofilms during chronic infection. There are a range of in vivo models that simulate chronic infections, such as surface wounds [75,76], subcutaneous wounds [77,78], implant-related (such as catheter, orthopaedic, and dental) [79–83], otitis media [84–86], and CF [87–90] to name but a few. As with all models, some are deemed more applicable than others. For instance, porcine models lend themselves greatly to the study of chronic wound infections due to similarities of the immune response systems, spatial structuring within tissue, and wound healing processes (re-epithelialisation, scarring, and tissue granulation) [91,92]. The complexities within in vitro biofilms such as structure, gene regulation, and the production of virulence factors, has been elucidated for many problematic opportunistic pathogens. However, during in vivo chronic infection, there is a complex interplay between host and pathogen, which is not easily replicated in vitro, and leads to observable differences between in vitro and in vivo “chronic infections”.

The in vivo biofilm differs from its in vitro counterpart, in both size and “shape”. A meta-analysis on the size of in vivo biofilms from chronic infections by Bjarnsholt and colleagues (2013) [20], showed them to have an upper size limit of 200 μm, which can be superseded in the presence of abiotic surfaces (e.g. catheters) leading to biofilms in excess of 1000 μm. Such observations differ drastically to the swathes of biofilm growth (up to several cm2) observed using in vitro methods [49]. It is thought that size limits placed on in vivo biofilms are the result of limiting factors, once thought to be nutrient based, but evidence points to oxygen depletion in the local environment [93]. In addition, biofilm “shape”, or more accurately the 3D structure, is different under in vitro conditions, with the characteristic mushroom structures of P. aeruginosa biofilm formation, yet to be observed in vivo.

Following periods of trauma, the natural microbial flora may develop into antagonistic biofilms and a state of chronicity. In vivo models of chronic infection require artificial inoculation, usually at an inflated concentration, and with the aid of foreign bodies, so that the inoculation is not cleared by the host immune system [82,94,95].

Another difference between many in vivo studies and actual in vivo infections, is the potential for multiple species to be present within the latter. A meta-analysis of 454 wound biofilms by Peters and colleagues from diabetic patients identified more than 1600 unique bacterial species, with diversity similar to that of the patients natural skin flora [35,96]. The presence of two or more unique bacterial species is observed in more than 80% of wounds analysed, with a relatively large proportion (30%) containing five or more species. Other studies place diversity higher, with an average of 5.4 bacterial species per chronic wound [36]. Such degrees of species diversity are also observed in the CF lung, resulting from the diverse microenvironments that arise [97–100]. Conversely, Kirketerp-Moller and colleagues (2008) [5] struggled to identify multiple species during a venous leg ulcer study, highlighting the diversity and complexity of different types of chronic infection. For instance, P. aeruginosa is known to grow in the deeper recesses of chronic leg ulcers, compared to S. aureus, which is found on the surface [42]. This non-random distribution suggests some species, especially anaerobes in deep recesses, will be missed if the wrong sampling method is employed. Therefore, an all-encompassing, biopsy-based extraction and molecular identification, should be employed to allow a thorough investigation of species within an infection site.

This degree of diversity within an infection site may result in a range of species-species interactions that have the potential to be both beneficial and antagonistic. For instance, Haemophilus influenzae and Moraxella catarrhalis grown concomitantly in an otitis media Chinchilla model, show an increased tolerance towards antimicrobial agents and the host’s response system [101]. In other models, differences are also observed. For instance, co-colonisation of two common CF pathogens (P. aeruginosa and Burkholderia cenocepacia) in a murine CF model leads to a mutualistic relationship, whereby P. aeruginosa persists, and B. cenocepacia alters the inflammatory response [102]. In some instances, the presence of multiple species within a specified environment is a necessity. For instance, during colonization of the oral cavity, primary colonisers (such as Candida albicans) allow the attachment of other species (such as Streptococcus spp.), facilitating the temporal co-aggregation of cells [103]. Should one of the primary colonisers be absent, co-aggregation will cease. Whilst highlighting the need to study multi-species infections, it also suggests that we need to incorporate a diverse range of species into experiments. It is important to re-iterate that the presence of multiple species within a system (such as an infection or test tube) does not mean they form a multi-species biofilm, which has not been observed to date. Therefore, it may be best to think about such systems as “battlefields”, with species not directly mixing, but residing within their own ecological space.

Whilst some species act synergistically with others through metabolite and signal production, and/or direct contact, some species have been observed to diversify without the need for multi-species interactions. For instance, during CF lung infection, P. aeruginosa gradually evolves, adapting to the CF lung through structural and dynamic changes over time that reduce virulence, and increase chronicity through a phenotypically heterogeneous population [104]. Interestingly, seemingly homogenous subpopulations of P. aeruginosa in the CF lung show large variation for many key aspects of infection, such as antibiotic tolerance, QS and virulence factor production [105–108]. Such diversity between and within species, may alter the pathogenicity of the infection, infection persistence, impact on the host and antibiotic resistance and tolerance.

To better understand the effects of biofilm infections on wound healing, both murine and rodent models have been devised [109,110], however there remains a disparity in their effectiveness due to the differences seen during wound healing in humans. As mentioned previously, porcine models can provide increased translational data for delayed wound healing in light of biofilm infection due to similar dermal properties (re-epithelialization, scarring, hair follicle placement and abundance), however their accessibility is more restrictive and expensive than that of murine/rodent models [111]. Implant models are a great way to blur the lines between in vitro and in vivo biofilms through the presence of an abiotic surface. Comparing in vivo models where an abiotic surface is present to in vitro biofilms, results in very similar biofilms (size, shape, and thickness) [20]. There are a range of in vivo models for chronic lung infection, however most require the require the bacterial cells to be well established on agar/agarose sheets or attached to the surface of alginate beads [95]. A murine model whereby cells are inhaled, has been developed to circumvent the need for various inoculation implants, whilst concomitantly allowing the upper and lower respiratory tract to be investigated [112]. Further to this, and more importantly, the model allows for persistence and adaptation of cells, similar to that observed in chronic infections, which includes evolutionary dynamics. The range of in vivo methods, along with their pros and cons have been summarised in Table 2.

View this table:
Table 2.

Comparison of different model systems for studying biofilms in vivo.

In vivo conditions, in vitro methods

Sometimes it may not be ethical, practicable, or feasible to conduct in vivo experimentation. Given the issues highlighted previously, how can we better represent in vivo conditions in our in vitro models? It is widely known, for P. aeruginosa at least, that different nutritional cues result in altered biofilm formation, virulence, motility, and QS [46,113–117]. These differences become increasingly important when factors of clinical relevance, such as virulence and antimicrobial tolerance, are altered [13,118,119]. Nutritional cues similar to those of expectorated CF sputum have been incorporated into a synthetic CF sputum media (SCFM) that approximates P. aeruginosa gene expression to that observed in expectorated CF sputum [120]. Two noteworthy points of this study are (i) the lack of key components observed in CF sputum (notably DNA, fatty acids, N-acetyle glucosamine, and mucin) and (ii) the inability of the methods used (RNA-seq) to correctly predict fitness requirements [121–127]. A follow up study rectified these issues, incorporating these components into SCFM. Using Tn-seq (which is a more precise way to measure fitness, compared to RNA-seq), it was shown that near identical selection pressures exist between synthetic and expectorated CF sputum [128]. In a separate study, the use of an artificial sputum media was shown to increase diversity within a population, something not observed in Lysogeny Broth (LB) [108]. This diversity increased in the presence of certain antibiotics at sub-inhibitory concentrations, highlighting the need for effective clearing of cells. Whilst the use of specialized media will not replace in vivo techniques, any way in which we can manipulate cells to generate increased diversity as seen during in vivo experiments, may result in findings that have increased clinical relevance.

Similarities have been observed in the active biosynthetic pathways of P. aeruginosa in the CF lung and murine surgical wound infections [127]. This suggests that (i) catabolite metabolism is shared between certain infection sites and (ii) other factors, such as host-inflammatory responses, may be the cause of infection chronicity. In light of this, it might be better to think of SCFM, not as a synthetic sputum media, but a synthetic infection media (SIM), which could then be supplemented further to better replicate the nutrient environment of in vivo conditions. For example, surface wounds contain high concentrations of both host-derived serum proteins and the fibrous extracellular matrix protein; collagen [129,130]. The presence of such compounds in growth media is known to reduce biofilm formation for a range of clinically relevant organisms, including P. aeruginosa and S. aureus [131–133]. The problem with such observations is that the experiments were performed using the microtitre assay, which does not take into account the possibility that attachment (to abiotic surfaces) might be the cause of such observations, and which under many in vivo conditions has little relevance. Such observations may also suggest why surface attachment (to biotic surfaces) is not generally observed in vivo, and why complex 3D structures do not develop.

In vitro techniques are widely criticized for their incorporation of abiotic surfaces, which only have clinical relevance to a small number of implant-related infections [67–69]. However, the widespread use of abiotic surfaces is not surprising due to the difficulties of trying to mimic the complex multicellular topology of an in vivo surface. The easiest way to negate these complexities is to employ surface independent methodologies, which have been shown to produce biofilms of equal size, shape, and antimicrobial tolerances to those observed in vivo [62,133,134]. Whilst these methods produce in vivo biofilms under in vitro conditions, they lack the complex 3D topology and spatial structure of host tissue, which will alter a range of factors from antimicrobial to nutrient and oxygen penetration [21,135,136].

Negative effects asserted on biofilm formation by serum proteins are also observed for antibiotic penetration in tissue samples [136]. As mentioned previously, bacteria are capable of occupying different ecological niches within a wound environment [42]. Growth of different methicillin-resistant Staphylococcus aureus (MRSA) strains on porcine nasal epithelium tissue resulted in three different growth profiles, highlighting the need to select strains with clinical relevance [137]. A large oversight of some in vitro biofilms which relate to CF lung infections, is the use of the P. aeruginosa reference strain PAO1. The lack of alginate production in PAO1 makes it difficult to gauge the relevance of these studies to CF lung infections. Whilst non-mucoid strains might be present in the CF lung, infection with P. aeruginosa is characterized in many instances by the presence of this mucoid phenotype. However, as one of the most studied P. aeruginosa strains, its use has been exemplary in identifying and understanding complex regulatory pathways. If we are to use clinical isolates more readily in experiments, these must be chosen with care. Recent studies have shown that in CF infections, although there may be only one infection clone of P. aeruginosa (eg: The Liverpool Epidemic Strain, LES), the population of LES can be highly phenotypically diverse [105,138]. More recently, population analysis of the LES, has identified the commonality of divergent sub-lineages and their co-existence, allowing them to exchange potentially adaptive mutations [138]. Put simply, which of these LES isolates would you select to be your choice as ‘the’ LES strain? In this instance, one option is to consider working with ‘populations’ of P. aeruginosa taken from a sputum sample rather than taking a single colony from a plate. We also wish to highlight the clinical ramifications of such diversity, especially during antimicrobial therapy, whereby sub-inhibitory concentrations can drive diversification which may affect the patients clinical outcome [108]. The extensive use of antibiotic “cocktails” during CF-lung related infections highlights the need to take diversification of populations seriously, and not rely on a single clone or strain. Combining this information, it might be more relevant to utilise clinical isolates from early infections, in a representative media with a clinical antibiotic treatment regime, thus allowing us to study the diversification process and how it impacts various aspects of disease.

If we wish to mimic the in vivo environment, then the use of ex vivo samples provides us with new opportunities [92,137,139–142]. By using ex vivo tissue samples with in vitro methods and synthetic media, it may be possible to create controlled environments similar to those seen in vivo. Whilst many factors, most importantly an immune response, remain absent, the replacement of an abiotic surface with a biotic one will allow the growth and study of biofilms similar to those seen in vivo. Each of these systems which attempt to bridge the knowledge gap between in vivo conditions and in vitro methods are summarised in Table 3.

View this table:
Table 3.

Methods that better represent in vivo conditions during in vitro investigation

Conclusion and recommendations

To date, most of our mechanistic knowledge and hypotheses surrounding biofilm formation and how this relates to chronic infection is based upon in vitro observations, primarily through the use of microtitre plate assays and flow cell systems. These systems have greatly enhanced our knowledge about the mechanisms of how cells attach to surfaces and differentiate into multicellular biofilms. However, it is becoming increasingly apparent that many of our in vitro methods do not accurately represent in vivo conditions, and so may provide only limited information that has any clinical relevance. Whilst this should come as no surprise, due to the innate complexity of biological systems, it highlights the need for better representation of the in vivo condition during in vitro investigation. At present, many in vitro studies contain (i) unrepresentative nutrients, (ii) uncharacteristic nutrient flow, (iii) uncharacteristic surfaces, and (iv) unrepresentative microorganisms. Whilst there is no “gold-standard” for the study of in vivo and in vitro biofilm formation, it is crucial to know the limiting factors of selected models so as to not over-extrapolate data, and generate assumptions beyond the capabilities of the model.

It is known that different nutrients alter the way in which bacterial cells grow. The use of synthetic media, which better represents the composition of in vivo exudate should be considered to produce data of increased clinical relevance. Other aspects of an in vivo micro-environment are more difficult to manipulate, such as the creation of oxygen gradients and a host immune response.

As a research community, we have examined mono-species cultures extensively, systematically, yet not exhaustively, and whilst a leap into polymicrobial research might seem counter intuitive (when multiple aspects of mono-culture are not fully understood), it presents itself as the next logical step. With the majority of chronic infections harboring polymicrobial communities, interactions between different species may critically influence various factors associated with chronic infection such as virulence and AMR. Another issue that we raised is the use of reference strains. Such strains present researchers with a well-defined system which creates some reproducibility of research between laboratories. However, the relevance of these strains to isolates taken directly from infection is often unclear and we should consider whether there is always a clinical relevance to work performed using reference strains. We also know that clinical isolates taken from the same patient can show considerable phenotypic diversity and so there may be a need to use ‘populations’ of bacteria taken from an infection to more accurately study biofilms and infection.

Rome was not built in a day. We cannot expect immediate paradigm shifts in the way experiments are performed, nor do we know how to either. Precise experiments using well defined media and reference strains remain of considerable importance in elucidating mechanisms which might be important for biofilm formation and virulence during infection. The use of representative species, in conjunction with other representative species (creating a poly-microbial “infection”), in media that is representative (such as synthetic sputum media), with representative surfaces (such as an ex vivo tissue sample), will likely produce data that is more relevant to in vivo infections. Finally, as biologists, we should not be afraid of performing, and being more accepting of, ‘dirty’ experiments. These are experiments where we have less control and stringency, and where some aspects are not 100% standardised. We suggest that this would be a positive way forward in helping us to understand the biology of infection better.

Acknowledgements

This work was funded by a Human Frontier Science grant to SPD and TB (RGY0081/2012).

Footnotes

  • ↵* Joint first authors

References

  1. ↵
    Costerton JW, Geesey GG, Cheng KJ. How bacteria stick. Sci Am 1978;238:86–95.
    OpenUrlCrossRefPubMedWeb of Science
  2. ↵
    McCoy WF, Bryers JD, Robbins J, Costerton JW. Observations of fouling biofilm formation. Can J Microbiol 1981;27:910–7.
    OpenUrlCrossRefPubMedWeb of Science
  3. ↵
    Bjarnsholt T, Jensen PØ, Fiandaca MJ, Pedersen J, Hansen CR, Andersen CB, et al. Pseudomonas aeruginosa biofilms in the respiratory tract of cystic fibrosis patients. Pediatr Pulmonol 2009;44:547–58. doi:10.1002/ppul.21011.
    OpenUrlCrossRefPubMed
  4. ↵
    Homøe P, Bjarnsholt T, Wessman M, Sørensen HCF, Johansen HK. Morphological evidence of biofilm formation in Greenlanders with chronic suppurative otitis media. Eur Arch Otorhi-nolaryngol 2009;266:1533–8. doi:10.1007/s00405-009-0940-9.
    OpenUrlCrossRefPubMed
  5. ↵
    Kirketerp-Møller K, Jensen PØ, Fazli M, Madsen KG, Pedersen J, Moser C, et al. Distribution, organization, and ecology of bacteria in chronic wounds. J Clin Microbiol 2008;46:2717–22. doi:10.1128/JCM.00501-08.
    OpenUrlAbstract/FREE Full Text
  6. Parsa K, Schaudinn C, Gorur A, Sedghizadeh PP, Johnson T, Tse DT, et al. Demonstration of bacterial biofilms in culture-negative silicone stent and jones tube. Ophthal Plast Reconstr Surg 2010;26:426–30. doi:10.1097/IOP.0b013e3181cff61f.
    OpenUrlCrossRefPubMed
  7. ↵
    Stoodley P, Ehrlich GD, Sedghizadeh PP, Hall-Stoodley L, Baratz ME, Altman DT, et al. Orthopaedic biofilm infections. Curr Orthop Pract 2011;22:558–63. doi:10.1097/BCO. 0b013e318230efcf.
    OpenUrlCrossRefPubMed
  8. ↵
    Murray CJ, Lopez AD. Alternative projections of mortality and disability by cause 1990–2020: Global Burden of Disease Study. Lancet 1997;349:1498–504. doi:10.1016/S0140-6736(96)07492-2.
    OpenUrlCrossRefPubMedWeb of Science
  9. ↵
    Lazarus GS, Cooper DM, Knighton DR, Margolis DJ, Pecoraro RE, Rodeheaver G, et al. Definitions and guidelines for assessment of wounds and evaluation of healing. Arch Dermatol 1994;130:489–93.
    OpenUrlCrossRefPubMedWeb of Science
  10. ↵
    Høiby N, Bjarnsholt T, Givskov M, Molin S, Ciofu O. Antibiotic resistance of bacterial biofilms. Int J Antimicrob Agents 2010;35:322–32. doi:10.1016/j.ijantimica0.2009.12.011.
    OpenUrlCrossRefPubMed
  11. Nelson R. Antibiotic development pipeline runs dry. New drugs to fight resistant organisms are not being developed, experts say. Lancet 2003;362:1726–7.
    OpenUrlCrossRefPubMedWeb of Science
  12. Rodríguez-Martínez JM, Pascual A. [Activity of antimicrobial agents on bacterial biofilms]. Enferm Infecc Microbiol Clin 2008;26:107–14.
    OpenUrlPubMed
  13. ↵
    Stewart PS, Costerton JW. Antibiotic resistance of bacteria in biofilms. Lancet 2001;358:135–8.
    OpenUrlCrossRefPubMedWeb of Science
  14. ↵
    Costerton JW. Bacterial Biofilms: A Common Cause of Persistent Infections. Science (80-) 1999;284:1318–22. doi:10.1126/science.284.5418.1318.
    OpenUrlAbstract/FREE Full Text
  15. ↵
    Wolcott R, Dowd S. The role of biofilms: are we hitting the right target? Plast Reconstr Surg 2011; 127 Suppl:280–350. doi:10.1097/PRS.0b013e3181fca244.
    OpenUrlCrossRef
  16. ↵
    O’Neill J. Antimicrobial Resistance: Tackling a crisis for the health and wealth of nations. Rev Antimicrob Resist 2014.
  17. ↵
    Høiby N, Bjarnsholt T, Moser C, Bassi GL, Coenye T, Donelli G, et al. ESCMID guideline for the diagnosis and treatment of biofilm infections 2014. Clin Microbiol Infect 2015;21 Suppl 1:S1–25. doi:10.1016/j.cmi.2014.10.024.
    OpenUrlCrossRef
  18. ↵
    Alhede M, Bjarnsholt T, Jensen PØ, Phipps RK, Moser C, Christophersen L, et al. Pseudomonas aeruginosa recognizes and responds aggressively to the presence of polymorphonuclear leukocytes. Microbiology 2009;155:3500–8. doi:10.1099/mic.0.031443-0.
    OpenUrlCrossRefPubMedWeb of Science
  19. ↵
    Bjarnsholt T, Jensen PØ, Burmølle M, Hentzer M, Haagensen JAJ, Hougen HP, et al. Pseudomonas aeruginosa tolerance to tobramycin, hydrogen peroxide and polymorphonuclear leukocytes is quorum-sensing dependent. Microbiology 2005;151:373–83. doi:10.1099/mic.0.27463-0.
    OpenUrlCrossRefPubMedWeb of Science
  20. ↵
    Bjarnsholt T, Alhede M, Alhede M, Eickhardt-Sørensen SR, Moser C, Kühl M, et al. The in vivo biofilm. Trends Microbiol 2013;21:466–74. doi:10.1016/j.tim.2013.06.002.
    OpenUrlCrossRefPubMedWeb of Science
  21. ↵
    Kragh KN, Alhede M, Jensen PØ, Moser C, Scheike T, Jacobsen CS, et al. Polymorphonuclear Leukocytes Restrict the Growth of Pseudomonas aeruginosa in the Lungs of Cystic Fibrosis Patients. Infect Immun 2014. doi:10.1128/IAI.01969-14.
    OpenUrlAbstract/FREE Full Text
  22. ↵
    Johnson GM, Lee DA, Regelmann WE, Gray ED, Peters G, Quie PG. Interference with granulocyte function by Staphylococcus epidermidis slime. Infect Immun 1986;54:13–20.
    OpenUrlAbstract/FREE Full Text
  23. ↵
    Costerton JW, Stewart PS, Greenberg EP. Bacterial biofilms: a common cause of persistent infections. Science 1999;284:1318–22.
    OpenUrlAbstract/FREE Full Text
  24. ↵
    Jesaitis AJ, Franklin MJ, Berglund D, Sasaki M, Lord CI, Bleazard JB, et al. Compromised host defense on Pseudomonas aeruginosa biofilms: characterization of neutrophil and biofilm interactions. J Immunol 2003;171:4329–39.
    OpenUrlAbstract/FREE Full Text
  25. ↵
    Van Gennip M, Christensen LD, Alhede M, Phipps R, Jensen PØ, Christophersen L, et al. Inactivation of the rhlA gene in Pseudomonas aeruginosa prevents rhamnolipid production, disabling the protection against polymorphonuclear leukocytes. APMIS 2009;117:537–46. doi:10.1111/j.1600-0463.2009.02466.x.
    OpenUrlCrossRefPubMedWeb of Science
  26. ↵
    Folsom JP, Richards L, Pitts B, Roe F, Ehrlich GD, Parker A, et al. Physiology of Pseudomonas aeruginosa in biofilms as revealed by transcriptome analysis. BMC Microbiol 2010;10:294. doi:10.1186/1471-2180-10-294.
    OpenUrlCrossRefPubMed
  27. ↵
    Mulcahy H, Charron-Mazenod L, Lewenza S. Extracellular DNA chelates cations and induces antibiotic resistance in Pseudomonas aeruginosa biofilms. PLoS Pathog 2008;4:e1000213. doi:10.1371/journal.ppat.1000213.
    OpenUrlCrossRefPubMed
  28. Pamp SJ, Sternberg C, Tolker-Nielsen T. Insight into the microbial multicellular lifestyle via flow-cell technology and confocal microscopy. Cytometry A 2009;75:90–103. doi:10.1002/cyto.a.20685.
    OpenUrlCrossRefPubMed
  29. ↵
    Stoodley P, Sauer K, Davies DG, Costerton JW. Biofilms as complex differentiated communities. Annu Rev Microbiol 2002;56:187–209. doi:10.1146/annurev.micro.56.012302.160705.
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    Chiang W-C, Nilsson M, Jensen PØ, Høiby N, Nielsen TE, Givskov M, et al. Extracellular DNA shields against aminoglycosides in Pseudomonas aeruginosa biofilms. Antimicrob Agents Chemother 2013;57:2352–61. doi:10.1128/AAC.00001-13.
    OpenUrlAbstract/FREE Full Text
  31. ↵
    De Beer D, Stoodley P, Lewandowski Z. Measurement of local diffusion coefficients in biofilms by microinjection and confocal microscopy. Biotechnol Bioeng 1997;53:151–8. doi: 10.1002/(SICI)1097-0290(19970120)53:2<151::AID-BIT4>3.0.CO;2-N.
    OpenUrlCrossRefPubMedWeb of Science
  32. ↵
    Stewart PS. Biophysics of biofilm infection. Pathog Dis 2013;70:212–8. doi: 10.1111/2049-632X.12118.
    OpenUrlCrossRef
  33. ↵
    Pamp SJ, Gjermansen M, Johansen HK, Tolker-Nielsen T. Tolerance to the antimicrobial peptide colistin in Pseudomonas aeruginosa biofilms is linked to metabolically active cells, and depends on the pmr and mexAB-oprM genes. Mol Microbiol 2008;68:223–40. doi:10.1111/j. 1365-2958.2008.06152.x.
    OpenUrlCrossRefPubMedWeb of Science
  34. ↵
    Walters MC, Roe F, Bugnicourt A, Franklin MJ, Stewart PS. Contributions of antibiotic penetration, oxygen limitation, and low metabolic activity to tolerance of Pseudomonas aeruginosa biofilms to ciprofloxacin and tobramycin. Antimicrob Agents Chemother 2003;47:317–23. doi:10.1128/AAC.47.1.317.
    OpenUrlAbstract/FREE Full Text
  35. ↵
    Peters BM, Jabra-Rizk MA, O’May GA, Costerton JW, Shirtliff ME. Polymicrobial interactions: impact on pathogenesis and human disease. Clin Microbiol Rev 2012;25:193–213. doi: 10.1128/CMR.00013-11.
    OpenUrlAbstract/FREE Full Text
  36. ↵
    Thomsen TR, Aasholm MS, Rudkjøbing VB, Saunders AM, Bjarnsholt T, Givskov M, et al. The bacteriology of chronic venous leg ulcer examined by culture-independent molecular methods. Wound Repair Regen 18:38–49. doi:10.1111/j.1524-475X.2009.00561.x.
    OpenUrlCrossRefPubMedWeb of Science
  37. ↵
    Rendueles O, Ghigo J-M. Multi-species biofilms: how to avoid unfriendly neighbors. FEMS Microbiol Rev 2012;36:972–89. doi:10.1111/j.1574-6976.2012.00328.x.
    OpenUrlCrossRefPubMedWeb of Science
  38. Elias S, Banin E. Multi-species biofilms: living with friendly neighbors. FEMS Microbiol Rev 2012;36:990–1004. doi:10.1111/j.1574-6976.2012.00325.x.
    OpenUrlCrossRefPubMedWeb of Science
  39. ↵
    Burmølle M, Ren D, Bjarnsholt T, Sørensen SJ. Interactions in multispecies biofilms: do they actually matter? Trends Microbiol 2014;22:84–91. doi:10.1016/j.tim.2013.12.004.
    OpenUrlCrossRefPubMedWeb of Science
  40. ↵
    Kolenbrander PE, Andersen RN, Moore L V. Intrageneric coaggregation among strains of human oral bacteria: potential role in primary colonization of the tooth surface. Appl Environ Microbiol 1990;56:3890–4.
    OpenUrlAbstract/FREE Full Text
  41. Murray JL, Connell JL, Stacy A, Turner KH, Whiteley M. Mechanisms of synergy in polymicrobial infections. J Microbiol 2014;52:188–99. doi:10.1007/s12275-014-4067-3.
    OpenUrlCrossRefPubMed
  42. ↵
    Fazli M, Bjarnsholt T, Kirketerp-Møller K, Jørgensen B, Andersen AS, Krogfelt KA, et al. Nonrandom distribution of Pseudomonas aeruginosa and Staphylococcus aureus in chronic wounds. J Clin Microbiol 2009;47:4084–9. doi:10.1128/JCM.01395-09.
    OpenUrlAbstract/FREE Full Text
  43. ↵
    McNab R, Ford SK, El-Sabaeny A, Barbieri B, Cook GS, Lamont RJ. LuxS-based signaling in Streptococcus gordonii: autoinducer 2 controls carbohydrate metabolism and biofilm formation with Porphyromonas gingivalis. J Bacteriol 2003;185:274–84.
    OpenUrlAbstract/FREE Full Text
  44. Waters CM, Bassler BL. Quorum sensing: cell-to-cell communication in bacteria. Annu Rev Cell Dev Biol 2005;21:319–46. doi:10.1146/annurev.cellbio.21.012704.131001.
    OpenUrlCrossRefPubMedWeb of Science
  45. ↵
    Federle MJ. Autoinducer-2-based chemical communication in bacteria: complexities of interspecies signaling. Contrib Microbiol 2009;16:18–32. doi:10.1159/000219371.
    OpenUrlCrossRefPubMed
  46. ↵
    Klausen M, Aaes-Jørgensen A, Molin S, Tolker-Nielsen T. Involvement of bacterial migration in the development of complex multicellular structures in Pseudomonas aeruginosa biofilms. Mol Microbiol 2003;50:61–8.
    OpenUrlCrossRefPubMedWeb of Science
  47. ↵
    Klausen M, Heydorn A, Ragas P, Lambertsen L, Aaes-Jørgensen A, Molin S, et al. Biofilm formation by Pseudomonas aeruginosa wild type, flagella and type IV pili mutants. Mol Microbiol 2003;48:1511–24.
    OpenUrlCrossRefPubMedWeb of Science
  48. ↵
    Lawrence JR, Korber DR, Hoyle BD, Costerton JW, Caldwell DE. Optical sectioning of microbial biofilms. J Bacteriol 1991;173:6558–67.
    OpenUrlAbstract/FREE Full Text
  49. ↵
    Barken KB, Pamp SJ, Yang L, Gjermansen M, Bertrand JJ, Klausen M, et al. Roles of type IV pili, flagellum-mediated motility and extracellular DNA in the formation of mature multicellular structures in Pseudomonas aeruginosa biofilms. Environ Microbiol 2008;10:2331–43. doi: 10.1111/j.1462-2920.2008.01658.x.
    OpenUrlCrossRefPubMedWeb of Science
  50. ↵
    Christensen GD, Simpson WA, Younger JJ, Baddour LM, Barrett FF, Melton DM, et al. Adherence of coagulase-negative staphylococci to plastic tissue culture plates: a quantitative model for the adherence of staphylococci to medical devices. J Clin Microbiol 1985;22:996–1006.
    OpenUrlAbstract/FREE Full Text
  51. ↵
    Pitts B, Hamilton MA, Zelver N, Stewart PS. A microtiter-plate screening method for biofilm disinfection and removal. J Microbiol Methods 2003;54:269–76.
    OpenUrlCrossRefPubMedWeb of Science
  52. Ceri H, Olson ME, Stremick C, Read RR, Morck D, Buret A. The Calgary Biofilm Device: new technology for rapid determination of antibiotic susceptibilities of bacterial biofilms. J Clin Microbiol 1999;37:1771–6.
    OpenUrlAbstract/FREE Full Text
  53. Das JR, Bhakoo M, Jones M V, Gilbert P. Changes in the biocide susceptibility of Staphylococcus epidermidis and Escherichia coli cells associated with rapid attachment to plastic surfaces. J Appl Microbiol 1998;84:852–8.
    OpenUrlCrossRefPubMed
  54. ↵
    Jakobsen TH, van Gennip M, Phipps RK, Shanmugham MS, Christensen LD, Alhede M, et al. Ajoene, a sulfur-rich molecule from garlic, inhibits genes controlled by quorum sensing. Antimicrob Agents Chemother 2012;56:2314–25. doi:10.1128/AAC.05919-11.
    OpenUrlAbstract/FREE Full Text
  55. ↵
    Campoccia D, Cangini I, Selan L, Vercellino M, Montanaro L, Visai L, et al. An overview of the methodological approach to the in vitro study of anti-infective biomaterials. Int J Artif Organs 2012;35:800–16. doi:10.5301/ijao.5000140.
    OpenUrlCrossRef
  56. ↵
    Goeres DM, Loetterle LR, Hamilton MA, Murga R, Kirby DW, Donlan RM. Statistical assessment of a laboratory method for growing biofilms. Microbiology 2005;151:757–62. doi: 10.1099/mic.0.27709-0.
    OpenUrlCrossRefPubMedWeb of Science
  57. ↵
    Goeres DM, Hamilton MA, Beck NA, Buckingham-Meyer K, Hilyard JD, Loetterle LR, et al. A method for growing a biofilm under low shear at the air-liquid interface using the drip flow biofilm reactor. Nat Protoc 2009;4:783–8. doi:10.1038/nprot.2009.59.
    OpenUrlCrossRefPubMed
  58. ↵
    Martinez-Gutierrez F, Boegli L, Agostinho A, Sánchez EM, Bach H, Ruiz F, et al. Anti-biofilm activity of silver nanoparticles against different microorganisms. Biofouling 2013;29:651–60. doi:10.1080/08927014.2013.794225.
    OpenUrlCrossRefPubMed
  59. Rudney JD, Chen R, Lenton P, Li J, Li Y, Jones RS, et al. A reproducible oral microcosm biofilm model for testing dental materials. J Appl Microbiol 2012;113:1540–53. doi: 10.1111/j.1365-2672.2012.05439.x.
    OpenUrlCrossRefPubMedWeb of Science
  60. ↵
    Zelver N, Hamilton M, Pitts B, Goeres D, Walker D, Sturman P, et al. Measuring antimicrobial effects on biofilm bacteria: from laboratory to field. Methods Enzymol 1999;310:608–28.
    OpenUrlCrossRefPubMed
  61. ↵
    Bjarnsholt T, Tolker-Nielsen T, Givskov M, Janssen M, Christensen LH. Detection of bacteria by fluorescence in situ hybridization in culture-negative soft tissue filler lesions. Dermatol Surg 2009;35 Suppl 2:1620–4. doi:10.1111/j.1524-4725.2009.01313.x.
    OpenUrlCrossRefPubMed
  62. ↵
    Alhede M, Kragh KN, Qvortrup K, Allesen-Holm M, van Gennip M, Christensen LD, et al. Phenotypes of non-attached pseudomonas aeruginosa aggregates resemble surface attached biofilm. PLoS One 2011;6.
  63. ↵
    Schleheck D, Barraud N, Klebensberger J, Webb JS, McDougald D, Rice SA, et al. Pseudomonas aeruginosa PAO1 preferentially grows as aggregates in liquid batch cultures and disperses upon starvation. PLoS One 2009;4:e5513. doi:10.1371/journal.pone.0005513.
    OpenUrlCrossRefPubMed
  64. ↵
    James GA, Swogger E, Wolcott R, Pulcini E deLancey, Secor P, Sestrich J, et al. Biofilms in chronic wounds. Wound Repair Regen 16:37–44. doi:10.1111/j.1524-475X.2007.00321.x.
    OpenUrlCrossRefPubMedWeb of Science
  65. ↵
    Bjarnsholt T, Kirketerp-Møller K, Jensen PØ, Madsen KG, Phipps R, Krogfelt K, et al. Why chronic wounds will not heal: a novel hypothesis. Wound Repair Regen 16:2–10. doi: 10.1111/ j.1524-475X.2007.00283.x.
    OpenUrlCrossRef
  66. ↵
    Kolpen M, Hansen CR, Bjarnsholt T, Moser C, Christensen LD, van Gennip M, et al. Polymorphonuclear leucocytes consume oxygen in sputum from chronic Pseudomonas aeruginosa pneumonia in cystic fibrosis. Thorax 2010;65:57–62. doi:10.1136/thx.2009.114512.
    OpenUrlAbstract/FREE Full Text
  67. ↵
    Costerton JW, Post JC, Ehrlich GD, Hu FZ, Kreft R, Nistico L, et al. New methods for the detection of orthopedic and other biofilm infections. FEMS Immunol Med Microbiol 2011;61:133–40. doi:10.1111/j.1574-695X.2010.00766.x.
    OpenUrlCrossRefPubMed
  68. ↵
    Nickel JC, Grant SK, Costerton JW. Catheter-associated bacteriuria. An experimental study. Urology 1985;26:369–75.
    OpenUrlCrossRefPubMedWeb of Science
  69. ↵
    Waar K, Degener JE, van Luyn MJ, Harmsen HJM. Fluorescent in situ hybridization with specific DNA probes offers adequate detection of Enterococcus faecalis and Enterococcus faecium in clinical samples. J Med Microbiol 2005;54:937–44. doi:10.1099/jmm.0.46022-0.
    OpenUrlCrossRefPubMedWeb of Science
  70. ↵
    Pogorelov AG, Chebotar I V., Pogorelova VN. Scanning electron microscopy of biofilms adherent to the inner catheter surface. Bull Exp Biol Med 2014;157:711–4. doi:10.1007/s10517-014-2648-0.
    OpenUrlCrossRef
  71. ↵
    Monds RD, O’Toole GA. The developmental model of microbial biofilms: ten years of a paradigm up for review. Trends Microbiol 2009;17:73–87. doi:10.1016/j.tim.2008.11.001.
    OpenUrlCrossRefPubMedWeb of Science
  72. ↵
    Tolker-Nielsen T, Sternberg C. Growing and analyzing biofilms in flow chambers. Curr Protoc Microbiol 2011;Chapter 1:Unit 1B.2. doi:10.1002/9780471729259.mc01b02s21.
    OpenUrlCrossRef
  73. ↵
    Høiby N, Ciofu O, Johansen HK, Song Z, Moser C, Jensen PØ, et al. The clinical impact of bacterial biofilms. Int J Oral Sci 2011;3:55–65. doi:10.4248/IJOS11026.
    OpenUrlCrossRefPubMed
  74. ↵
    Vyas KS, Wong LK. Detection of Biofilm in Wounds as an Early Indicator for Risk for Tissue Infection and Wound Chronicity. Ann Plast Surg 2015. doi:10.1097/SAP.0000000000000440.
    OpenUrlCrossRef
  75. ↵
    Akiyama H, Kanzaki H, Tada J, Arata J. Staphylococcus aureus infection on cut wounds in the mouse skin: experimental staphylococcal botryomycosis. J Dermatol Sci 1996;11:234–8.
    OpenUrlCrossRefPubMedWeb of Science
  76. ↵
    Dai T, Tegos GP, Zhiyentayev T, Mylonakis E, Hamblin MR. Photodynamic therapy for methicillin-resistant Staphylococcus aureus infection in a mouse skin abrasion model. Lasers Surg Med 2010;42:38–44. doi:10.1002/lsm.20887.
    OpenUrlCrossRefPubMedWeb of Science
  77. ↵
    Davis SC, Ricotti C, Cazzaniga A, Welsh E, Eaglstein WH, Mertz PM. Microscopic and physiologic evidence for biofilm-associated wound colonization in vivo. Wound Repair Regen 16:23–9. doi:10.1111/j.1524-475X.2007.00303.x.
    OpenUrlCrossRefPubMedWeb of Science
  78. ↵
    Engelsman AF, van Dam GM, van der Mei HC, Busscher HJ, Ploeg RJ. In vivo evaluation of bacterial infection involving morphologically different surgical meshes. Ann Surg 2010;251:133–7. doi:10.1097/SLA.0b013e3181b61d9a.
    OpenUrlCrossRefPubMed
  79. ↵
    Guiton PS, Hung CS, Hancock LE, Caparon MG, Hultgren SJ. Enterococcal biofilm formation and virulence in an optimized murine model of foreign body-associated urinary tract infections. Infect Immun 2010;78:4166–75. doi:10.1128/IAI.00711-10.
    OpenUrlAbstract/FREE Full Text
  80. Kadurugamuwa JL, Modi K, Yu J, Francis KP, Purchio T, Contag PR. Noninvasive biophotonic imaging for monitoring of catheter-associated urinary tract infections and therapy in mice. Infect Immun 2005;73:3878–87. doi:10.1128/IAI.73.7.3878-3887.2005.
    OpenUrlAbstract/FREE Full Text
  81. Fung LCT, Mittelman MW, Thorner PS, Khoury AE. A novel rabbit model for the evaluation of biomaterial associated urinary tract infection. Can J Urol 2003;10:2007–12.
    OpenUrlPubMed
  82. ↵
    Kadurugamuwa JL, Sin L, Albert E, Yu J, Francis K, DeBoer M, et al. Direct continuous method for monitoring biofilm infection in a mouse model. Infect Immun 2003;71:882–90.
    OpenUrlAbstract/FREE Full Text
  83. ↵
    Rimondini L, Farè S, Brambilla E, Felloni A, Consonni C, Brossa F, et al. The effect of surface roughness on early in vivo plaque colonization on titanium. J Periodontol 1997;68:556–62. doi:10.1902/jop.1997.68.6.556.
    OpenUrlCrossRefPubMed
  84. ↵
    Chaney EJ, Nguyen CT, Boppart SA. Novel method for non-invasive induction of a middleear biofilm in the rat. Vaccine 2011;29:1628–33. doi:10.1016/j.vaccine.2010.12.076.
    OpenUrlCrossRefPubMedWeb of Science
  85. Eriksson P-O, Li J, Ny T, Hellström S. Spontaneous development of otitis media in plasminogen-deficient mice. Int J Med Microbiol 2006;296:501–9. doi:10.1016/j.ijmm.2006.04.002.
    OpenUrlCrossRefPubMed
  86. ↵
    Prabhakara R, Harro JM, Leid JG, Harris M, Shirtliff ME. Murine immune response to a chronic Staphylococcus aureus biofilm infection. Infect Immun 2011;79:1789–96. doi: 10.1128/IAI.01386-10.
    OpenUrlAbstract/FREE Full Text
  87. ↵
    Cash HA, Woods DE, McCullough B, Johanson WG, Bass JA. A rat model of chronic respiratory infection with Pseudomonas aeruginosa. Am Rev Respir Dis 1979;119:453–9.
    OpenUrlPubMedWeb of Science
  88. Clarke LL, Grubb BR, Gabriel SE, Smithies O, Koller BH, Boucher RC. Defective epithelial chloride transport in a gene-targeted mouse model of cystic fibrosis. Science 1992;257:1125–8.
    OpenUrlAbstract/FREE Full Text
  89. Pedersen SS, Shand GH, Hansen BL, Hansen GN. Induction of experimental chronic Pseudomonas aeruginosa lung infection with P. aeruginosa entrapped in alginate microspheres. APMIS 1990;98:203–11.
    OpenUrlCrossRefPubMedWeb of Science
  90. ↵
    Starke JR, Edwards MS, Langston C, Baker CJ. A mouse model of chronic pulmonary infection with Pseudomonas aeruginosa and Pseudomonas cepacia. Pediatr Res 1987;22:698–702. doi:10.1203/00006450-198712000-00017.
    OpenUrlCrossRefPubMedWeb of Science
  91. ↵
    Sullivan TP, Eaglstein WH, Davis SC, Mertz P. The pig as a model for human wound healing. Wound Repair Regen 9:66–76.
  92. ↵
    Harrison F, Muruli A, Higgins S, Diggle SP. Development of an ex vivo porcine lung model for studying growth, virulence, and signaling of Pseudomonas aeruginosa. Infect Immun 2014;82:3312–23. doi:10.1128/IAI.01554-14.
    OpenUrlAbstract/FREE Full Text
  93. ↵
    Costerton JW, Lewandowski Z, DeBeer D, Caldwell D, Korber D, James G. Biofilms, the customized microniche. J Bacteriol 1994;176:2137–42.
    OpenUrlFREE Full Text
  94. ↵
    Christensen LD, Moser C, Jensen PØ, Rasmussen TB, Christophersen L, Kjelleberg S, et al. Impact of Pseudomonas aeruginosa quorum sensing on biofilm persistence in an in vivo intraperitoneal foreign-body infection model. Microbiology 2007;153:2312–20. doi:10.1099/ mic.0.2007/006122-0.
    OpenUrlCrossRefPubMedWeb of Science
  95. ↵
    Kukavica-Ibrulj I, Levesque RC. Animal models of chronic lung infection with Pseudomonas aeruginosa: useful tools for cystic fibrosis studies. Lab Anim 2008;42:389–412. doi:10.1258/la.2007.06014e.
    OpenUrlCrossRefPubMed
  96. ↵
    Percival SL, Emanuel C, Cutting KF, Williams DW. Microbiology of the skin and the role of biofilms in infection. Int Wound J 2012;9:14–32. doi:10.1111/j.1742-481X.2011.00836.x.
    OpenUrlCrossRefPubMedWeb of Science
  97. ↵
    Rogers GB, Hart CA, Mason JR, Hughes M, Walshaw MJ, Bruce KD. Bacterial diversity in cases of lung infection in cystic fibrosis patients: 16S ribosomal DNA (rDNA) length heterogeneity PCR and 16S rDNA terminal restriction fragment length polymorphism profiling. J Clin Microbiol 2003;41:3548–58.
    OpenUrlAbstract/FREE Full Text
  98. Rogers GB, Carroll MP, Serisier DJ, Hockey PM, Jones G, Bruce KD. characterization of bacterial community diversity in cystic fibrosis lung infections by use of 16s ribosomal DNA terminal restriction fragment length polymorphism profiling. J Clin Microbiol 2004;42:5176–83. doi:10.1128/JCM.42.11.5176-5183.2004.
    OpenUrlAbstract/FREE Full Text
  99. Sibley CD, Parkins MD, Rabin HR, Duan K, Norgaard JC, Surette MG. A polymicrobial perspective of pulmonary infections exposes an enigmatic pathogen in cystic fibrosis patients. Proc Natl Acad Sci U S A 2008;105:15070–5. doi:10.1073/pnas.0804326105.
    OpenUrlAbstract/FREE Full Text
  100. ↵
    Zhao J, Schloss PD, Kalikin LM, Carmody LA, Foster BK, Petrosino JF, et al. Decade-long bacterial community dynamics in cystic fibrosis airways. Proc Natl Acad Sci U S A 2012;109:5809–14. doi:10.1073/pnas.1120577109.
    OpenUrlAbstract/FREE Full Text
  101. ↵
    Armbruster CE, Hong W, Pang B, Weimer KED, Juneau RA, Turner J, et al. Indirect pathogenicity of Haemophilus influenzae and Moraxella catarrhalis in polymicrobial otitis media occurs via interspecies quorum signaling. MBio 2010;1. doi:10.1128/mBio.00102-10.
    OpenUrlAbstract/FREE Full Text
  102. ↵
    Bragonzi A, Farulla I, Paroni M, Twomey KB, Pirone L, Lorè NI, et al. Modelling co-infection of the cystic fibrosis lung by Pseudomonas aeruginosa and Burkholderia cenocepacia reveals influences on biofilm formation and host response. PLoS One 2012;7:e52330. doi: 10.1371/journal.pone.0052330.
    OpenUrlCrossRefPubMed
  103. ↵
    Rickard AH, Gilbert P, High NJ, Kolenbrander PE, Handley PS. Bacterial coaggregation: an integral process in the development of multi-species biofilms. Trends Microbiol 2003;11:94–100.
    OpenUrlCrossRefPubMedWeb of Science
  104. ↵
    Sousa AM, Pereira MO. Pseudomonas aeruginosa Diversification during Infection Development in Cystic Fibrosis Lungs-A Review. Pathog (Basel, Switzerland) 2014;3:680–703. doi: 10.3390/pathogens3030680.
    OpenUrlCrossRefPubMed
  105. ↵
    Darch SE, McNally A, Harrison F, Corander J, Barr HL, Paszkiewicz K, et al. Recombination is a key driver of genomic and phenotypic diversity in a Pseudomonas aeruginosa population during cystic fibrosis infection. Sci Rep 2015;5:7649. doi:10.1038/srep07649.
    OpenUrlCrossRefPubMed
  106. Hall AJ, Fothergill JL, McNamara PS, Southern KW, Winstanley C. Turnover of strains and intraclonal variation amongst Pseudomonas aeruginosa isolates from paediatric CF patients. Diagn Microbiol Infect Dis 2014;80:324–6. doi:10.1016/j.diagmicrobio.2014.09.007.
    OpenUrlCrossRef
  107. Mowat E, Paterson S, Fothergill JL, Wright EA, Ledson MJ, Walshaw MJ, et al. Pseudomonas aeruginosa population diversity and turnover in cystic fibrosis chronic infections. Am J Respir Crit Care Med 2011;183:1674–9. doi:10.1164/rccm.201009-14300C.
    OpenUrlCrossRefPubMedWeb of Science
  108. ↵
    Wright EA, Fothergill JL, Paterson S, Brockhurst MA, Winstanley C. Sub-inhibitory concentrations of some antibiotics can drive diversification of Pseudomonas aeruginosa populations in artificial sputum medium. BMC Microbiol 2013;13:170. doi: 10.1186/1471-2180-13-170.
    OpenUrlCrossRefPubMed
  109. ↵
    Dalton T, Dowd SE, Wolcott RD, Sun Y, Watters C, Griswold JA, et al. An In Vivo Polymicrobial Biofilm Wound Infection Model to Study Interspecies Interactions. PLoS One 2011;6:e27317. doi:10.1371/journal.pone.0027317.
    OpenUrlCrossRefPubMed
  110. ↵
    DeLeon S, Clinton A, Fowler H, Everett J, Horswill AR, Rumbaugh KP. Synergistic interactions of Pseudomonas aeruginosa and Staphylococcus aureus in an in vitro wound model. Infect Immun 2014;82:4718–28. doi:10.1128/IAI.02198-14.
    OpenUrlAbstract/FREE Full Text
  111. ↵
    Roche ED, Renick PJ, Tetens SP, Ramsay SJ, Daniels EQ, Carson DL. Increasing the presence of biofilm and healing delay in a porcine model of MRSA-infected wounds. Wound Repair Regen 20:537–43. doi:10.1111/j.1524-475X.2012.00808.x.
    OpenUrlCrossRef
  112. ↵
    Fothergill JL, Neill DR, Loman N, Winstanley C, Kadioglu A. Pseudomonas aeruginosa adaptation in the nasopharyngeal reservoir leads to migration and persistence in the lungs. Nat Commun 2014;5:4780. doi:10.1038/ncomms5780.
    OpenUrlCrossRef
  113. ↵
    Banin E, Vasil ML, Greenberg EP. Iron and Pseudomonas aeruginosa biofilm formation. Proc Natl Acad Sci U S A 2005;102:11076–81. doi:10.1073/pnas.0504266102.
    OpenUrlAbstract/FREE Full Text
  114. Palmer KL, Aye LM, Whiteley M. Nutritional cues control Pseudomonas aeruginosa multicellular behavior in cystic fibrosis sputum. J Bacteriol 2007;189:8079–87. doi:10.1128/JB.01138-07.
    OpenUrlAbstract/FREE Full Text
  115. Rietsch A, Wolfgang MC, Mekalanos JJ. Effect of metabolic imbalance on expression of type III secretion genes in Pseudomonas aeruginosa. Infect Immun 2004;72:1383–90.
    OpenUrlAbstract/FREE Full Text
  116. Santos AS, Sampaio APW, Vasquez GS, Santa Anna LM, Pereira N, Freire DMG. Evaluation of different carbon and nitrogen sources in production of rhamnolipids by a strain of Pseudomonas aeruginosa. Appl Biochem Biotechnol 2002;98-100:1025–35.
    OpenUrlCrossRef
  117. ↵
    Shrout JD, Chopp DL, Just CL, Hentzer M, Givskov M, Parsek MR. The impact of quorum sensing and swarming motility on Pseudomonas aeruginosa biofilm formation is nutritionally conditional. Mol Microbiol 2006;62:1264–77. doi:10.1111/j.1365-2958.2006.05421.x.
    OpenUrlCrossRefPubMedWeb of Science
  118. ↵
    Drenkard E, Ausubel FM. Pseudomonas biofilm formation and antibiotic resistance are linked to phenotypic variation. Nature 2002;416:740–3. doi:10.1038/416740a.
    OpenUrlCrossRefPubMedWeb of Science
  119. ↵
    Son MS, Matthews WJ, Kang Y, Nguyen DT, Hoang TT. In vivo evidence of Pseudomonas aeruginosa nutrient acquisition and pathogenesis in the lungs of cystic fibrosis patients. Infect Immun 2007;75:5313–24. doi:10.1128/IAI.01807-06.
    OpenUrlAbstract/FREE Full Text
  120. ↵
    Palmer KL, Mashburn LM, Singh PK, Whiteley M. Cystic fibrosis sputum supports growth and cues key aspects of Pseudomonas aeruginosa physiology. J Bacteriol 2005;187:5267–77. doi:10.1128/JB.187.15.5267-5277.2005.
    OpenUrlAbstract/FREE Full Text
  121. ↵
    Brandt T, Breitenstein S, von der Hardt H, Tümmler B. DNA concentration and length in sputum of patients with cystic fibrosis during inhalation with recombinant human DNase. Thorax 1995;50:880–2.
    OpenUrlAbstract/FREE Full Text
  122. Deutschbauer A, Price MN, Wetmore KM, Shao W, Baumohl JK, Xu Z, et al. Evidence-based annotation of gene function in Shewanella oneidensis MR-1 using genome-wide fitness profiling across 121 conditions. PLoS Genet 2011;7:e1002385. doi:10.1371/journal.pgen.1002385.
    OpenUrlCrossRefPubMed
  123. Fung C, Naughton S, Turnbull L, Tingpej P, Rose B, Arthur J, et al. Gene expression of Pseudomonas aeruginosa in a mucin-containing synthetic growth medium mimicking cystic fibrosis lung sputum. J Med Microbiol 2010;59:1089–100. doi:10.1099/jmm.0.019984-0.
    OpenUrlCrossRefPubMedWeb of Science
  124. Hull J, South M, Phelan P, Grimwood K. Surfactant composition in infants and young children with cystic fibrosis. Am J Respir Crit Care Med 1997;156:161–5. doi:10.1164/ajrccm. 156.1.9609090.
    OpenUrlCrossRefPubMedWeb of Science
  125. Korgaonkar AK, Whiteley M. Pseudomonas aeruginosa enhances production of an antimicrobial in response to N-acetylglucosamine and peptidoglycan. J Bacteriol 2011;193:909–17. doi: 10.1128/JB.01175-10.
    OpenUrlAbstract/FREE Full Text
  126. Mayer C, Moritz R, Kirschner C, Borchard W, Maibaum R, Wingender J, et al. The role of intermolecular interactions: studies on model systems for bacterial biofilms. Int J Biol Macromol 1999;26:3–16.
    OpenUrlCrossRefPubMedWeb of Science
  127. ↵
    Turner KH, Everett J, Trivedi U, Rumbaugh KP, Whiteley M. Requirements for Pseudomonas aeruginosa acute burn and chronic surgical wound infection. PLoS Genet 2014;10:e1004518. doi:10.1371/journal.pgen.1004518.
    OpenUrlCrossRefPubMed
  128. ↵
    Turner KH, Wessel AK, Palmer GC, Murray JL, Whiteley M. Essential genome of Pseudomonas aeruginosa in cystic fibrosis sputum. Proc Natl Acad Sci U S A 2015;112:4110–5. doi:10.1073/pnas.1419677112.
    OpenUrlAbstract/FREE Full Text
  129. ↵
    Cullen B, Watt PW, Lundqvist C, Silcock D, Schmidt RJ, Bogan D, et al. The role of oxidised regenerated cellulose/collagen in chronic wound repair and its potential mechanism of action. Int J Biochem Cell Biol 2002;34:1544–56.
    OpenUrlCrossRefPubMedWeb of Science
  130. ↵
    Lehnhardt M, Jafari HJ, Druecke D, Steinstraesser L, Steinau HU, Klatte W, et al. A qualitative and quantitative analysis of protein loss in human burn wounds. Burns 2005;31:159–67. doi:10.1016/j.burns.2004.08.015.
    OpenUrlCrossRefPubMedWeb of Science
  131. ↵
    Abraham NM, Jefferson KK. A low molecular weight component of serum inhibits biofilm formation in Staphylococcus aureus. Microb Pathog 2010;49:388–91. doi:10.1016/j.micpath. 2010.07.005.
    OpenUrlCrossRefPubMed
  132. Ding X, Liu Z, Su J, Yan D. Human serum inhibits adhesion and biofilm formation in Candida albicans. BMC Microbiol 2014;14:80. doi:10.1186/1471-2180-14-80.
    OpenUrlCrossRef
  133. ↵
    Werthén M, Henriksson L, Jensen PØ, Sternberg C, Givskov M, Bjarnsholt T. An in vitro model of bacterial infections in wounds and other soft tissues. APMIS 2010; 118:156–64. doi: 10.1111/j.1600-0463.2009.02580.x.
    OpenUrlCrossRefPubMed
  134. ↵
    Crone S, Garde C, Bjarnsholt T, Alhede M. A novel in vitro wound biofilm model used to evaluate low-frequency ultrasonic-assisted wound debridement. J Wound Care 2015;24:64, 66–9, 72. doi:10.12968/jowc.2015.24.2.64.
    OpenUrlCrossRef
  135. ↵
    Bergan T, Engeset A, Olszewski W. Does serum protein binding inhibit tissue penetration of antibiotics? Rev Infect Dis 9:713–8.
  136. ↵
    Nix DE, Goodwin SD, Peloquin CA, Rotella DL, Schentag JJ. Antibiotic tissue penetration and its relevance: impact of tissue penetration on infection response. Antimicrob Agents Chemother 1991;35:1953–9.
    OpenUrlFREE Full Text
  137. ↵
    Tulinski P, Fluit AC, van Putten JPM, de Bruin A, Glorieux S, Wagenaar JA, et al. An ex vivo porcine nasal mucosa explants model to study MRSA colonization. PLoS One 2013;8:e53783. doi:10.1371/journal.pone.0053783.
    OpenUrlCrossRef
  138. ↵
    Williams D, Evans B, Haldenby S, Walshaw MJ, Brockhurst MA, Winstanley C, et al. Divergent, Coexisting Pseudomonas aeruginosa Lineages in Chronic Cystic Fibrosis Lung Infections. Am J Respir Crit Care Med 2015;191:775–85. doi:10.1164/rccm.201409-1646OC.
    OpenUrlCrossRefPubMed
  139. ↵
    Carterson AJ, Höner zu Bentrup K, Ott CM, Clarke MS, Pierson DL, Vanderburg CR, et al. A549 lung epithelial cells grown as three-dimensional aggregates: alternative tissue culture model for Pseudomonas aeruginosa pathogenesis. Infect Immun 2005;73:1129–40. doi: 10.1128/IAI.73.2.1129-1140.2005.
    OpenUrlAbstract/FREE Full Text
  140. Chuang-Smith ON, Wells CL, Henry-Stanley MJ, Dunny GM. Acceleration of Enterococcus faecalis biofilm formation by aggregation substance expression in an ex vivo model of cardiac valve colonization. PLoS One 2010;5:e15798. doi:10.1371/journal.pone.0015798.
    OpenUrlCrossRefPubMed
  141. Huang T-Y, Gulabivala K, Ng Y-L. A bio-molecular film ex-vivo model to evaluate the influence of canal dimensions and irrigation variables on the efficacy of irrigation. Int Endod J 2008;41:60–71. doi:10.1111/j.1365-2591.2007.01317.x.
    OpenUrlCrossRefPubMed
  142. ↵
    Wolcott RD, Rumbaugh KP, James G, Schultz G, Phillips P, Yang Q, et al. Biofilm maturity studies indicate sharp debridement opens a time-dependent therapeutic window. J Wound Care 2010;19:320–8. doi:10.12968/jowc.2010.19.8.77709.
    OpenUrlCrossRefPubMedWeb of Science
  143. Sternberg C, Christensen BB, Johansen T, Toftgaard Nielsen A, Andersen JB, Givskov M, et al. Distribution of bacterial growth activity in flow-chamber biofilms. Appl Environ Microbiol 1999;65:4108–17.
    OpenUrlAbstract/FREE Full Text
  144. Johnson AW, Sidman JD, Lin J. Bioluminescent imaging of pneumococcal otitis media in chinchillas. Ann Otol Rhinol Laryngol 2013;122:344–52.
    OpenUrlCrossRefPubMed
  145. Palmer J. Bacterial biofilms in chronic rhinosinusitis. Ann Otol Rhinol Laryngol Suppl 2006;196:35–9.
    OpenUrlPubMed
Back to top
Posted November 27, 2015.
Download PDF
Email
The limitations of in vitro experimentation in understanding biofilms and chronic infection
Aled E. L. Roberts, Kasper N. Kragh, Thomas Bjarnsholt, Stephen P. Diggle
bioRxiv 032987; doi: https://doi.org/10.1101/032987
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area