On the approximate realization of continuous mappings by neural networks

doi:10.1016/0893-6080(89)90003-8

Neural Networks

Volume 2, Issue 3, 1989, Pages 183-192

https://doi.org/10.1016/0893-6080(89)90003-8 Get rights and content

Abstract

In this paper, we prove that any continuous mapping can be approximately realized by Rumelhart-Hinton-Williams' multilayer neural networks with at least one hidden layer whose output functions are sigmoid functions. The starting point of the proof for the one hidden layer case is an integral formula recently proposed by Irie-Miyake and from this, the general case (for any number of hidden layers) can be proved by induction. The two hidden layers case is proved also by using the Kolmogorov-Arnold-Sprecher theorem and this proof also gives non-trivial realizations.

References (18)

Y. Uesaka
Analog perceptrons: On additive representation of functions
Information and Control
(1971)
S. Amari
A theory of adaptive pattern classifiers
IEEE Transactions on Electronic Computers
(1967)
R.O. Duda et al.
Pattern classification by iteratively determined linear and piecewise linear discriminant functions
IEEE Transactions on Electronic Computers
(1966)
I.M. Gel'fand et al.
R. Hecht-Nielsen
Kolmogorov mapping neural network existence theorem
W.Y. Huang et al.
Neural net and traditional classifiers
B. Irie et al.
Capabilities of three-layered Perceptrons
A.N. Kolmogorov
On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition
Doklady Akademii Nauk SSSR
(1957)
American Mathematical Society Translation
(1963)
R.P. Lippmann
An introduction to computing with neural nets
IEEE ASSP Magazine
(1987, April)

There are more references available in the full text version of this article.

Cited by (3449)

Physics-driven neural networks for nonlinear micromechanics
2024, International Journal of Mechanical Sciences
Micromechanical problems seek full-field solutions in response to external and/or internal thermal-mechanical loads, which have been increasingly encountered in material property and process assessments. Here we present a machine learning based strategy of solving micromechanics at finite strains. Deep neural networks (DNNs) are trained by employing the elastic energy together with other physical constraints to formulate the loss function. In particular, the crossover of material points, which can be a common issue for similar DNNs when applied to compression-dominant problems, is shown to be effectively addressed by including a kinematic penalty term that forces the DNN to maintain structural integrity and stability and avoid unphysical behavior. The generality and accuracy of the proposed physics-driven neural networks (PDNNs) are demonstrated through various micromechanical problems, including single crystal anisotropy elasticity, Eshelby's inclusion problem, buckling, and elastic homogenization of polycrystals. The effect of the choice of optimizers and hyperparameters on the PDNN training are discussed and the computational efficiency is also analyzed. It is shown that this novel PDNN framework can completely remove the need of labeled training data and exhibit improved performances as compared to the conventional physics-informed neural networks (PINNs) in terms of avoiding unphysical solutions and attaining higher computational efficiency. PDNNs can thus serve as a promising tool to solve some critical challenges associated with a wide range of nonlinear micromechanical problems.
Prediction of critical temperature of binary refrigerant mixtures by neural network
2024, International Journal of Refrigeration
Critical temperature ( $T_{c}$ ) is an important property of refrigerant mixtures. In this study, a neural network-based model is established to predict the $T_{c}$ of binary mixtures from the molar fraction and molecular structure of their components. First, graph neural network (GNN) is applied to generate the representation of the molecular structure of single component. The generated representation is added according to the molar fraction as the input of the proposed model. In order to embed the prior knowledge, pre-trained parameters are reused in the proposed model to accelerate convergence and improve accuracy. A dataset that contains 66 binary mixtures and 505 data points is collected from reference to adjust the remaining model parameters. The performance of the obtained model is comprehensively evaluated and it shows good prediction accuracy and generalization ability. The average absolute relative deviation (AARD) in the total dataset is 0.80%. For most of the mixtures, the AARD is less than 1%. Finally, the proposed model is compared with three empirical methods and the equation of state (EOS)-based method in REFPROP, and the model in this study show satisfactory result and significant improved accuracy for some mixture types.
Fading memory as inductive bias in residual recurrent networks
2024, Neural Networks
Residual connections have been proposed as an architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increased task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs.
ML-based pre-deployment SDN performance prediction with neural network boosting regression
2024, Expert Systems with Applications
Software defined networking (SDN) has been proposed as an effective approach to improve network management efficiency and increase network intelligence in various networks. However, configuring and deploying suitable SDN functionalities is challenging. To overcome these obstacles in SDN network deployment, machine learning models have been introduced to monitor and predict the quality of service (QoS) in an SDN-enabled network. However, most existing studies are applicable only to deployed SDN networks with specific network topologies. In this study, a prediction-based SDN network configuration and deployment scheme is proposed for unseen network topologies before the actual deployment of SDN functionalities. In addition, a machine learning-based pre-deployment SDN performance prediction problem is formulated, and a neural network boosting regression model (i.e., NNBoost) is proposed as the solution. Numerical experiments demonstrate that NNBoost outperforms other machine learning algorithms, including deep learning methods, proposed in the literature, on datasets generated with both real-world and synthetic network topologies, compared with random forest (RF), AdaBoost, XGBoost, LightGBM, deep neural network (DNN) and long short-term memory (LSTM). It is found that NNBoost achieves lower root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE) results for the maximum and mean values for the Round-Trip Time (RTT), Switch-to-Controller (S2C) traffic, and Controller-to-Controller (C2C) traffic. It is also found that by adding extra training data samples generated with synthetic network topologies, NNBoost achieves a better prediction performance on the test data samples for real-world network topologies.
Physics-informed neural network integrate with unclosed mechanism model for turbulent mass transfer
2024, Chemical Engineering Science
Turbulent mass transfer is widespread in chemical engineering processes including separation, reaction, and others. Traditionally, turbulent mass transfer processes can be simulated by computational fluid dynamics (CFD) methods. However, CFD methods require adequate turbulent models for the fluid flow and mass (and heat) transfer, which are normally difficult to develop for reliable solutions. Moreover, the CFD simulation is computationally intensive and hard to use in the cases like process optimization or analysis where the simulation needs to be called repeatedly. In this paper, physics-informed neural network (PINN) is developed and trained by unclosed mechanism model and sparse observation data of turbulent mass transfer process. The PINN method shows stronger generalization ability in solving the velocity, pressure and concentration fields in a turbulent mass transfer process than traditional deep neural network (DNN) method and turbulent Schmidt model. Under different boundary conditions, the PINN developed in the present paper can instantly predict the concentration distributions with sufficient accuracy and be used for inverse computation for estimating the turbulent viscosity and mass diffusivity as output results. The PINN is also capable of handling data noise by adjusting parameters, suggesting its potential in integrating experimental data.
A deep difference collocation method and its application in elasticity problems
2024, International Journal of Solids and Structures
Based on the coordinate transformation and difference scheme, a deep difference collocation method (DDCM) that solves the partial differential equations (PDEs) via a square domain is proposed in this paper. In most of deep learning-based methods such as physics-informed neural networks (PINNs), boundary losses and calculation of the items of PDEs in a physical domain using automatic differentiation will result in time-consuming training. DDCM utilizes a difference approximation approach to expedite the computation of neural network derivatives, enhancing training efficiency. It is adaptable to irregular physical domains via coordinate transformation and provides versatility in enforcing boundary conditions, enabling partial or total elimination of boundary losses. This research covers the analysis of 2D and 3D elastic problems along with the Kirchhoff plate bending problem. The validity of the introduced method is verified by comparing the obtained results to those from theoretical solutions, Finite Element Method (FEM), and Boundary Element Method (BEM). In comparison with PINN, the proposed methodology demonstrates a notably enhanced computational efficiency in solving PDEs, and exhibits commendable stability under a specified computational memory constraint.

View all citing articles on Scopus

View full text

Original contributionOn the approximate realization of continuous mappings by neural networks

Abstract

Information and Control

A theory of adaptive pattern classifiers

IEEE Transactions on Electronic Computers

Pattern classification by iteratively determined linear and piecewise linear discriminant functions

IEEE Transactions on Electronic Computers

Kolmogorov mapping neural network existence theorem

Neural net and traditional classifiers

Capabilities of three-layered Perceptrons

On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition

Doklady Akademii Nauk SSSR

American Mathematical Society Translation

An introduction to computing with neural nets

IEEE ASSP Magazine

Original contribution
On the approximate realization of continuous mappings by neural networks