Callister, W. D. & Rethwisch, D. G. Materials science and engineering: an introduction. Vol. 7. (John wiley & sons New York, 2007).
Martin, J. D. What’s in a name change? Phys. Perspect. 17, 3–32 (2015).
Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).
Chen, B.-C. et al. Lattice light-sheet microscopy: Imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998 (2014).
Levi, A. F. & Aeppli, G. The Naked Chip: no trade secret or hardware trojan can hide from ptychographic X-ray laminography. IEEE Spectr. 59, 38–43 (2022).
Stevenson, A. W. et al. Phase-contrast X-ray imaging with synchrotron radiation for materials science applications. Nucl. Instrum. Methods Phys. Res. Sect. B Beam Interact. Mater. At. 199, 427–435 (2003).
Fan, C. & Zhao, Z. Synchrotron radiation in materials science: light sources, techniques, and applications (John Wiley & Sons, 2018).
Yao, Y. et al. AutoPhaseNN: unsupervised physics-aware deep learning of 3D nanoscale Bragg coherent diffraction imaging. npj Comput. Mater. 8, 124 (2022).
Lee, C.-H. et al. Deep learning enabled strain mapping of single-atom defects in two-dimensional transition metal dichalcogenides with sub-picometer precision. Nano Lett. 20, 3369–3377 (2020).
Chen, C.-C. et al. Three-dimensional imaging of dislocations in a nanoparticle at atomic resolution. Nature 496, 74–77 (2013).
Spence, J. C. The future of atomic resolution electron microscopy for materials science. Mater. Sci. Eng. R: Rep. 26, 1–49 (1999).
Mukherjee, D. et al. Atomic-scale measurement of polar entropy. Phys. Rev. B 100, 104102 (2019).
Lin, Y. et al. Analytical transmission electron microscopy for emerging advanced materials. Matter 4, 2309–2339 (2021).
Williams, D. B. & Carter, C. B. The transmission electron microscope (Springer, 1996)
Pennycook, S. J. & Nellist, P. D. Scanning transmission electron microscopy: imaging and analysis (Springer Science & Business Media, 2011).
Crewe, A. V. Scanning transmission electron microscopy. J. Microsc. 100, 247–259 (1974).
Nellist, P. & Pennycook, S. The principles and interpretation of annular dark-field Z-contrast imaging. In Advances in imaging and electron physics (Elsevier, 2000) p. 147–203.
Nakane, T. et al. Single-particle cryo-EM at atomic resolution. Nature 587, 152–156 (2020).
Cheng, Y., Grigorieff, N., Penczek, P. A. & Walz, T. A primer to single-particle cryo-electron microscopy. Cell 161, 438–449 (2015).
Ramasse, Q. M. Twenty years after: How “Aberration correction in the STEM” truly placed a “A synchrotron in a Microscope”. Ultramicroscopy 180, 41–51 (2017).
Haider, M., Uhlemann, S. & Zach, J. Upper limits for the residual aberrations of a high-resolution aberration-corrected STEM. Ultramicroscopy 81, 163–175 (2000).
Krivanek, O., et al. Aberration correction in the STEM, in Electron Microscopy and Analysis 1997. (CRC Press, 1997) p. 35–40.
Sawada, H., Sasaki, T., Hosokawa, F. & Suenaga, K. Atomic-resolution STEM imaging of graphene at low voltage of 30 kV with resolution enhancement by using large convergence angle. Phys. Rev. Lett. 114, 166102 (2015).
Konno, M. et al. Lattice imaging at an accelerating voltage of 30 kV using an in-lens type cold field-emission scanning electron microscope. Ultramicroscopy 145, 28–35 (2014).
Pennycook, S. J. The impact of STEM aberration correction on materials science. Ultramicroscopy 180, 22–33 (2017).
Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-angstrom resolution. Nature 559, 343-+ (2018).
Nelson, C. T. et al. Spontaneous vortex nanodomain arrays at ferroelectric heterointerfaces. Nano Lett. 11, 828–834 (2011).
Stone, G. et al. Atomic scale imaging of competing polar states in a Ruddlesden–Popper layered oxide. Nat. Commun. 7, 12572 (2016).
Mukherjee, D., Miao, L., Stone, G. & Alem, N. mpfit: a robust method for fitting atomic resolution images with multiple Gaussian peaks. Adv. Struct. Chem. Imaging 6, 1 (2020).
Chisholm, M. F., et al. Atomic-scale compensation phenomena at polar interfaces. Phys. Rev. Lett. 105, 197602 (2010).
Hong, Z. J. et al. Stability of polar vortex lattice in ferroelectric superlattices. Nano Lett. 17, 2246–2252 (2017).
Li, Q., et al. Quantification of flexoelectricity in PbTiO3/SrTiO3 superlattice polar vortices using machine learning and phase-field modeling. Nat. Commun. 8, 1468 (2017).
Borisevich, A. Y., et al. Exploring mesoscopic physics of vacancy-ordered systems through atomic scale observations of topological defects. Phys. Rev. Lett., 2012. 109, 065702 (2012).
Miao, L. et al. Double-Bilayer polar nanoregions and Mn antisites in (Ca, Sr)3Mn2O7. Nature. Communications 13, 4927 (2022).
Kim, T. H. et al. Polar metals by geometric design. Nature 533, 68–72 (2016).
Yadav, A. K. et al. Observation of polar vortices in oxide superlattices. Nature 530, 198–201 (2016).
Chen, Z. et al. Mixed-state electron ptychography enables sub-angstrom resolution imaging with picometer precision at low dose. Nat. Commun. 11, 2994 (2020).
Varela, M. et al. Spectroscopic imaging of single atoms within a bulk solid. Phys. Rev. Lett. 92, 095502 (2004).
Brown, L. A synchrotron in a microscope. In Electron Microscopy and Analysis (CRC Press, 1997) p. 17–22.
Mukherjee, D., Gamler, J. T. L., Skrabalak, S. E. & Unocic, R. R. Lattice strain measurement of core@Shell electrocatalysts with 4D scanning transmission electron microscopy nanobeam electron diffraction. ACS. Catalysis 10, 5529–5541 (2020).
Han, Y. et al. Strain mapping of two-dimensional heterostructures with subpicometer precision. Nano Lett. 18, 3746–3751 (2018).
Ophus, C. Four-dimensional scanning transmission electron microscopy (4D-STEM): from scanning nanodiffraction to ptychography and beyond. Microsc. Microanal. 25, 563–582 (2019).
Kirkland, E. J. Advanced computing in electron microscopy. Vol. 12. (Springer, 1998).
Bonnet, N. Artificial intelligence and pattern recognition techniques in microscope image processing and analysis. In Advances in Imaging and Electron Physics (Elsevier, 2000), p. 1–77.
Bonnet, N. Multivariate statistical methods for the analysis of microscope image series: applications in materials science. J. Microsc. 190, 2–18 (1998).
Kalinin, S. V., Sumpter, B. G. & Archibald, R. K. Big-deep-smart data in imaging for guiding materials design. Nat. Mater. 14, 973 (2015).
Jesse, S., et al. Big data analytics for scanning transmission electron microscopy ptychography. Sci. Rep. 6, 26348 (2016).
Ziatdinov, M. et al. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS Nano 11, 12742–12752 (2017).
Ziatdinov, M., et al. Data mining graphene: correlative analysis of structure and electronic degrees of freedom in graphenic monolayers with defects. Nanotechnology 27, 495703 (2016).
Kalinin, S. V. et al. Machine learning in scanning transmission electron microscopy. Nat. Rev. Methods Prim. 2, 11 (2022).
Schwartz, J. et al. Imaging atomic-scale chemistry from fused multi-modal electron microscopy. npj Comput. Mater. 8, 16 (2022).
Munshi, J. et al. Disentangling multiple scattering with deep learning: application to strain mapping from electron diffraction patterns. npj Comput. Mater. 8, 254 (2022).
Bruefach, A., Ophus, C. & Scott, M. C. Analysis of interpretable data representations for 4D-STEM using unsupervised learning. Microsc. Microanal. 28, 1998–2008 (2022).
Roccapriore, K. M. et al. Automated experiment in 4D-STEM: exploring emergent physics and structural behaviors. ACS Nano 16, 7605–7614 (2022).
Cherukara, M. J. et al. AI-enabled high-resolution scanning coherent diffraction imaging. Appl. Phys. Lett. 117, 044103 (2020).
Mukherjee, D. et al. A roadmap for edge computing enabled automated multidimensional transmission electron. Microsc. Microsc. Today 30, 10–19 (2022).
Cao, M. C., Chen, Z., Jiang, Y. & Han, Y. Automatic parameter selection for electron ptychography via Bayesian optimization. Sci. Rep. 12, 12284 (2022).
Bosman, M., Watanabe, M., Alexander, D. T. L. & Keast, V. J. Mapping chemical and bonding information using multivariate analysis of electron energy-loss spectrum images. Ultramicroscopy 106, 1024–1032 (2006).
Torruella, P. et al. Clustering analysis strategies for electron energy loss spectroscopy (EELS). Ultramicroscopy 185, 42–48 (2018).
Jesse, S. & Kalinin, S. V. Principal component and spatial correlation analysis of spectroscopic-imaging data in scanning probe microscopy. Nanotechnology 20, 085714 (2009).
Griffin, L. A., Gaponenko, I. & Bassiri-Gharb, N. Better, faster, and less biased machine learning: electromechanical switching in ferroelectric thin films. Adv. Mater. 32, 2002425 (2020).
Qin, S., Guo, Y., Kaliyev, A. T. & Agar, J. C. Why it is unfortunate that linear machine learning “works” so well in electromechanical switching of ferroelectric thin films. Adv. Mater. 34, 2202814 (2022).
Agar, J. C. et al. Revealing ferroelectric switching character using deep recurrent neural networks. Nat. Commun. 10, 4809 (2019).
Higgins, I., et al. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations (ICLR, 2017).
Kalinin, S. V. et al. Deep Bayesian local crystallography. Npj Comput. Mater. 7, 12 (2021).
Kalinin, S. V., Dyck, O., Jesse, S. & Ziatdinov, M. Exploring order parameters and dynamic processes in disordered systems via variational autoencoders. Sci. Adv. 7, eabd5084 (2021).
Kalinin, S. V. et al. Disentangling ferroelectric domain wall geometries and pathways in dynamic piezoresponse force microscopy via unsupervised machine learning. Nanotechnology 33, 11 (2022).
Ziatdinov, M., Maksov, A., & Kalinin, S. V. Learning surface molecular structures via machine vision. Npj Comput. Mater. 3, 31 (2017).
Doty, C. et al. Design of a graphical user interface for few-shot machine learning classification of electron microscopy data. Comput. Mater. Sci. 203, 111121 (2022).
Somnath, S. et al. USID and pycroscopy–Open source frameworks for storing and analyzing imaging and spectroscopy data. Microsc. Microanal. 25, 220–221 (2019).
Jesse, S. et al. Atomic-level sculpting of crystalline oxides: toward bulk nanofabrication with single atomic plane precision. Small 11, 5895–5900 (2015).
Unocic, R. R. et al. Direct-write liquid phase transformations with a scanning transmission electron microscope. Nanoscale 8, 15581–15588 (2016).
Dyck, O., Lupini, A. R. & Jesse, S. Atom-by-Atom Direct Writing. Nano. Letters 23, 2339–2346 (2023).
Sang, X. et al. Dynamic scan control in STEM: spiral scans. Adv. Struct. Chem. Imaging 2, 6 (2016).
Roccapriore, K. M. et al. Sculpting the plasmonic responses of nanoparticles by directed electron beam irradiation. Small 18, 10 (2022).
Al-Najjar, A., et al. Enabling autonomous electron microscopy for networked computation and steering. In 2022 IEEE 18th International Conference on e-Science (e-Science) (IEEE, 2022).
Casas Moreno, X. et al. An open-source microscopy framework for simultaneous control of image acquisition, reconstruction, and analysis. HardwareX 13, e00400 (2023).
Olszta, M. et al. An automated scanning transmission electron microscope guided by sparse data analytics. Microsc. Microanal. 28, 1611–1621 (2022).
Kalinin, S. V. et al. Automated and autonomous experiments in electron and scanning probe microscopy. ACS Nano 15, 12604–12627 (2021).
Schorb, M. et al. Software tools for automated transmission electron microscopy. Nat. Methods 16, 471–477 (2019).
Vasudevan, R. K., Ziatdinov, M., Jesse, S. & Kalinin, S. V. Phases and interfaces from real space atomically resolved data: physics-based deep data image analysis. Nano Lett. 16, 5574–5581 (2016).
Akers, S. et al. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Comput. Mater. 7, 187 (2021).
Lewis, N. R. et al. Forecasting of in situ electron energy loss spectroscopy. npj Comput. Mater. 8, 252 (2022).
Zhang, C. & Ma, Y. Ensemble machine learning: methods and applications (Springer Science & Business Media, 2012).
Ghosh, A., Sumpter, B. G., Dyck, O., Kalinin, S. V. & Ziatdinov, M. Ensemble learning-iterative training machine learning for uncertainty quantification and automated experiment in atom-resolved microscopy. npj Comput. Mater. 7, 100 (2021).
Roccapriore, K. M. et al. Probing electron beam induced transformations on a single-defect level via automated scanning transmission electron microscopy. ACS nano 16, 17116–17127 (2022).
Jacob Madsen, T. S. The abTEM code: transmission electron microscopy from first principles. Open Res. Eur. 1, 24 (2021).
Schwenker, E. et al. Ingrained-An automated framework for fusing atomic-scale image simulations into experiments. Small. 18(19), e2102960 (2022).
Lingerfelt, E. J. et al. BEAM: a computational workflow system for managing and modeling material characterization data in HPC environments. Proc. Comput. Sci. 80, 2276–2280 (2016).
Merz, K. M. Jr. et al. Method and data sharing and reproducibility of scientific results. J. Chem. Inf. Model. 60, 5868–5869 (2020).
Ghosh, A., Ziatdinov, M., Dyck, O., Sumpter, B. & Kalinin, S. V. Bridging microscopy with molecular dynamics and quantum simulations: an AtomAI based pipeline. npj Comput. Mater. 8, 11 (2021).
Champion, K., Lusch, B., Kutz, J. N. & Brunton, S. L. Data-driven discovery of coordinates and governing equations. Proc. Natl Acad. Sci. 116, 22445–22451 (2019).
Rudy, S. H., Brunton, S. L., Proctor, J. L. & Kutz, J. N. Data-driven discovery of partial differential equations. Sci. Adv. 3, e1602614 (2017).
Cottrill, A. L. et al. Simultaneous inversion of optical and infra-red image data to determine thermo-mechanical properties of thermally conductive solid materials. Int. J. Heat. Mass Transf. 163, 120445 (2020).
Liu, Y., Ziatdinov, M. & Kalinin, S. V. Exploring causal physical mechanisms via non-gaussian linear models and deep kernel learning: applications for ferroelectric domain structures. ACS Nano 16, 9 (2021).
Nelson, C. et al. Mapping causal patterns in crystalline solids. Preprint at https://arxiv.org/abs/2103.01951 (2021).
Ziatdinov, M. et al. Causal analysis of competing atomistic mechanisms in ferroelectric materials from high-resolution scanning transmission electron microscopy data. npj Comput. Mater. 6, 127 (2020).
Vasudevan, R. K., Ziatdinov M., Vlcek, L. & Kalinin, S. V. Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality. npj Comput. Mater. 7, 127 (2021).
Benedek, N. A. & Fennie, C. J., Hybrid improper ferroelectricity: a mechanism for strong polairzation-magnetization coupling. Phys. Rev. Lett. 106, 107204 (2011).
Mulder, A. T., Benedek, N. A., Rondinelli, J. M. & Fennie, C. J. Turning ABO3 antiferroelectrics into ferroelectrics: design rules for practical rotation-driven ferroelectricity in double perovskites and A3B2O7 Ruddlesden-Popper compounds. Adv. Funct. Mater. 23, 4810–4820 (2013).
Balachandran,P. V., Young, J., Lookman, T. & Rondinelli, J. M. Learning from data to design functional materials without inversion symmetry. Nat. Commun. 8, 14282 (2017).
Rondinelli, J. M. & Fennie, C. J. Octahedral rotation-induced ferroelectricity in cation ordered perovskites. Adv. Mater. 24, 1961–1968 (2012).
Ghosh, A., Palanichamy, G., Trujillo, D. P., Shaikh, M. & Ghosh, S. Insights into cation ordering of double perovskite oxides from machine learning and causal relations. Chem. Mater. 34, 7563–7578 (2022).
Fiedler, K. R., et al. Evaluating stage motion for automated electron microscopy. https://doi.org/10.1093/micmic/ozad108 (2022).
Roccapriore, K. M., Creange, N., Ziatdinov, M. & Kalinin, S. V. Identification and correction of temporal and spatial distortions in scanning transmission electron microscopy. Ultramicroscopy 229, 113337 (2021).
Ophus, C., Ciston, J. & Nelson, C. T. Correcting nonlinear drift distortion of scanning probe and scanning transmission electron microscopies from image pairs with orthogonal scan directions. Ultramicroscopy 162, 1–9 (2016).
Sang, X. & LeBeau, J. M. Revolving scanning transmission electron microscopy: correcting sample drift distortion without prior knowledge. Ultramicroscopy 138, 28–35 (2014).
Jones, L. & Nellist, P. D. Identifying and correcting scan noise and drift in the scanning transmission electron microscope. Microsc. Microanal. 19, 1050–1060 (2013).
Roccapriore, K. M., Kalinin, S. V. & Ziatdinov, M. Physics discovery in nanoplasmonic systems via autonomous experiments in scanning transmission electron microscopy. Adv. Sci. 9, 2203422 (2022).
Duarte, J. et al. Fast inference of deep neural networks in FPGAs for particle physics. J. Instrum. 13, P07027 (2018).
Team, F. Machine learning on FPGAs using HLS, https://github.com/fastmachinelearning/hls4ml (2023).
Umuroglu, Y. et al., Finn: a framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (ACM, 2017).
Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous systems, http://tensorflow.org/ (2015).
Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning library. In Advances in neural information processing systems (NeurIPS, 2019).
Al-Najjar, A. & Rao, N. S. V. Virtual infrastructure twin for computing-instrument ecosystems: software and measurements. IEEE Access 11, 20254–20266 (2023).
Karras, T., Laine, S. & Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (IEEE, 2020), p. 4401–4410.
