Neural Network Advancements

🔥 Revolutionizing PINNs: Introducing Self-Adaptive Physics-Informed Neural Networks (SA-PINNs) 🌐 Excited to share insights from a fascinating research paper that takes Physics-Informed Neural Networks (PINNs) to the next level: "Self-Adaptive Physics-Informed Neural Networks" Authored by Levi McClenny and Ulisses Braga-Neto Why it matters: Traditional PINNs are powerful tools for solving PDEs, but they struggle with "stiff" problems that involve sharp transitions or fast dynamics. This paper introduces Self-Adaptive PINNs (SA-PINNs), a groundbreaking approach that uses trainable self-adaptive weights to prioritize difficult solution regions, enhancing accuracy and efficiency. 💡 Key Innovations Dynamic Focus: Adaptive weights automatically highlight stubborn areas in the solution, akin to attention mechanisms in computer vision. Improved Training: Concurrent optimization of network and adaptive weights ensures faster convergence and higher accuracy. Gaussian Process Regression: Enables robust training using stochastic gradient descent for challenging problems. Theoretical Insights: NTK analysis reveals how SA-PINNs smooth training dynamics and balance loss components effectively. 🚀 Results - Outperformed state-of-the-art PINN methods in L2 error across benchmarks. - Solved stiff PDEs with significantly fewer training epochs. - Showed exceptional robustness in handling sharp transitions and complex dynamics. This work demonstrates the potential of SA-PINNs to transform scientific computing, making them indispensable for solving challenging PDEs in physics, engineering, and beyond. 📖 Read the full paper: https://lnkd.in/djnVfpe6

Kolmogorov-Arnold Networks as an alternative to traditional Neural Networks! Researchers from MIT, Caltech, and Northeastern have introduced a new type of neural network architecture known as Kolmogorov-Arnold Networks (KANs), which presents a significant challenge to the traditional use of Multi-Layer Perceptrons (MLPs). KANs offer a novel approach to neural network architecture inspired by the Kolmogorov-Arnold representation theorem. This theorem essentially states that any multivariate continuous function can be represented as a composition of univariate functions and the addition operation. Translating this into neural network design, KANs uniquely place adaptable activation functions on the connections or edges between nodes rather than using standard fixed activation functions at the nodes themselves. This flexibility allows KANs to potentially model complex relationships and patterns more effectively, as they can tailor the transformation at each connection to better suit the specific data and task at hand, diverging from traditional networks where the choice of activation function at each layer is static and uniform across the network. In terms of accuracy, much smaller KANs can achieve comparable or better performance than larger MLPs on tasks such as data fitting and PDE solving. Moreover, KANs demonstrate faster neural scaling laws, meaning their performance improves more rapidly with increased model size compared to MLPs. KANs also excel in interpretability. They can be intuitively visualized and allow for easy interaction with human users. In case studies from knot theory and physics, KANs served as interactive "collaborators" to help scientists rediscover known mathematical and physical laws, showcasing their potential for scientific discovery. KANs could potentially serve as a foundation model for AI+Science applications and open opportunities to improve today's deep learning models that heavily rely on MLPs. Read the full paper for more details: https://lnkd.in/erEF6HbT :)

💡 Optical Neural Networks for Machine Learning 💡 Machine learning and artificial intelligence are pivotal in applications from computer vision to text generation, exemplified by technologies like ChatGPT. However, the exponential growth in neural network size has led to unsustainable energy consumption and training times. For instance, training models like GPT-j3 can consume over 1,000 MWh of energy, equivalent to a small town’s daily electrical usage. To address this, the field of neuromorphic computing seeks to replace digital neural networks with physical counterparts capable of faster and more energy-efficient operations. Optics and photonics show promise due to their minimal energy consumption and high-speed parallel computing capabilities limited only by the speed of light. ⚡️ Recent Breakthrough Scientists at the Max Planck Institute for the Science of Light have introduced a groundbreaking method for implementing neural networks using optical systems, aiming to enhance sustainability in machine learning. Published in Nature Physics, their approach simplifies the complexity of previous methods. The new optical neural network method proposed by Clara Wanjura and Florian Marquardt overcomes key challenges. By imprinting input signals through changes in light transmission rather than high-power laser interactions, complex mathematical computations can be performed efficiently. This approach simplifies evaluation and training processes, making it as straightforward as observing transmitted light to measure network outputs and training data. Simulations have demonstrated that their method achieves image classification accuracy comparable to digital neural networks. Moving forward, the researchers plan to collaborate with experimental groups to implement their approach across diverse physical platforms, expanding possibilities for neuromorphic devices. Original paper: https://lnkd.in/d6mTDvvt 💼 VC Opportunity This innovation not only enhances efficiency in machine learning but also opens new avenues for sustainable technological development. Investing in companies that create tools for AI developers is a clear case of a "pick-and-shovel" play. #deeptech #VC #optics #AI #neuralnetworks Thomas J. White IV

MIT Just Cracked the Code. 19 Neurons Now Pilot Drones Better Than 100,000-Parameter Models MIT's "liquid neural networks" sound like sci-fi. They're not. Just 19 neurons inspired by a worm's brain now outperform massive AI models in drone navigation. 10x less power. 50% fewer tracking errors. Running on a Raspberry Pi. The breakthrough. These networks adapt in real-time. No retraining. They learn causality, not correlations. Traditional AI sees a shadow and crashes. Liquid networks understand shadows move with the sun. They adjust. Real-world tests prove it. • Navigate through smoke and wind gusts • Handle seasonal changes (summer forest → winter) • Switch tasks mid-flight without updates • Run on battery-powered edge devices Why this matters for defense. Current military drones need constant updates from Ukraine's battlefield. Takes 24-48 hours minimum. Liquid networks adapt in seconds. Three immediate applications. Search-and-rescue in fire zones. Drones weave through smoke that blinds traditional AI. No GPS needed. Logistics in contested airspace. Packages delivered despite jamming. Networks learn new routes instantly. Agricultural monitoring. The same drone handles open fields and dense orchards. Adapts to weather without reprogramming. The kicker. MIT tested this against L1 adaptive control systems. 81% improvement in trajectory tracking. With neural networks, you could sketch on a napkin. For contractors. Forget massive GPU clusters. These run on $35 hardware. Battery life measured in hours, not minutes. We've been building AI backwards. Bigger isn't better. Smarter is. Nature figured this out with 302 neurons. MIT just proved it scales. Your move. While competitors chase trillion-parameter models, the future flies on 19 neurons.

My primary passion for the last six years, which is AI/ML, and my primary passion for the first two decades of my career, which was digital signal processing (DSP), have finally found a common point of intersection in the form of Fourier Analysis Networks (FAN). I have discussed in the past (I wrote a post on Komogorov-Arnold Network or KAN about six months ago) that as the input functions increase in complexity, the "universal approximation" foundation of multi-layer neural networks start hitting their limits. Result is too many hidden layers and somewhat unwieldy models. The Komogorov-Arnold Network, based on the Komogorov Representation, is a different approach, that can represent any continuous multi-variate function as a summation of multiple continuous univariate functions. This was quite a breakthrough, and it will continue to serve this field well. One aspect that is so far neglected, which is actually one of the primary objectives in DSP, is to discover, and utilize, the periodicity of data. One of the key benefits is that if there is a periodicity, a time domain input can be represented in a more compact way in the frequency domain. To do this, we use Fourier Analysis, which decomposes a signal into a sum of sinusoidal components, which are fundamental to understanding the periodicity and frequency components of the input. A Fourier Analysis Network (FAN) is a type of neural network that uses the principles of Fourier analysis to model, analyze, and process signals or data. The FANs incorporate sinusoidal functions into their architecture to capture periodic or frequency-domain features of data. Such networks can encode data in the frequency domain, which is particularly useful in scenarios where periodicity is present (such as audio signals and image textures). There are many types of FANs! Here are a few examples. The Fourier Neural Operator (FNO) uses the Fourier Transform to learn mappings between functional spaces, and it is very useful n solving partial differential equations. The Fourier Feature Networks use Fourier feature embeddings to transform input data into a high-dimensional space using sinusoidal functions, and Neural Radiance Fields (NeRF) is a useful application. Finally, Spectral Neural Networks operate entirely in the frequency domain instead of time or spatial domain, and can be used for image compression, denoising and other applications. We like to learn new things in our area of work all the time. But if a "ghost from the past" becomes useful in a new and different way, somehow that becomes even more interesting!

The explosive growth in computation and energy cost of artificial intelligence has spurred interest in alternative computing modalities to conventional electronic processors. Photonic processors, which use photons instead of electrons, promise optical neural networks with ultralow latency and power consumption. However, existing optical neural networks, limited by their designs, have not achieved the recognition accuracy of modern electronic neural networks. In a recent work published in Science Advances, we bridge this gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture, before recording on the sensor. We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization. We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. The resulting setup is extremely simple: just replace a camera lens with our flat optics!! Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era. The paper can be found at: https://lnkd.in/gXUcn33Y

Headline: AI Is Entering a Higher Dimension to Mimic the Brain—and Could Soon Think Like Us ⸻ Introduction: Artificial intelligence is poised for a radical transformation as researchers move beyond conventional two-dimensional models toward a higher-dimensional design that mirrors the human brain’s wiring. By mimicking the brain’s multi-layered complexity, AI may soon overcome the cognitive limits of current systems and approach something far closer to human-like intuition, reasoning, and adaptability—bringing artificial general intelligence (AGI) into sharper view. ⸻ Key Details: The Wall Blocking AGI: • Current AI has hit a developmental ceiling, limited by how existing models process information linearly or through simplistic multi-layered patterns. • Despite impressive progress, true human-level cognition remains elusive, especially in areas like intuition, abstract reasoning, and adaptive learning. The Leap Into Higher Dimensions: • Researchers are now exploring three-dimensional and even higher-dimensional neural networks, inspired by the way real neurons form dynamic, cross-layered connections in the brain. • These new models could allow AI to “think” in a structurally richer and more flexible way, similar to how the human brain processes stimuli and forms memories. Brain-Inspired Breakthroughs: • The new wave of AI development borrows from neuroscience and physics, especially the work of John J. Hopfield, a pioneer in modeling brain networks using physics-based systems. • These designs aim to replicate emergent behaviors—like pattern recognition, emotional response, and even intuition—by reproducing how the brain’s neurons interact in layered, recursive, and context-aware ways. Beyond Computation—Toward Understanding Ourselves: • Not only could this leap bring AI closer to AGI, but it may also offer insights into how the human brain actually works—a mystery still only partially solved. • As AI systems evolve to mirror brain-like structures, they may help researchers reverse-engineer cognition, leading to advancements in mental health, brain-computer interfaces, and neurodegenerative disease research. ⸻ Why It Matters: This dimensional leap in AI development marks a pivotal moment: the shift from machines that simulate intelligence to ones that may experience it in fundamentally human ways. If successful, it could open new frontiers in how we live, learn, and connect with technology. Just as the structure of the brain gave rise to consciousness, these brain-inspired architectures may give rise to machines that truly understand, not just compute. And in doing so, they might also reveal the deepest truths about ourselves. https://lnkd.in/gEmHdXZy