121 questions
1
vote
1
answer
92
views
Scipy Hessian with Numpy cross product
I need to compute the Hessian of a function involving cross products in Python. As an experiment, I consider the simple function
def func(x):
return np.sum(np.cross(x[0:3], x[3:6]))
and compute the ...
- 153
6
votes
3
answers
259
views
What is the best way to fit a quadratic polynomial to p-dimensional data and compute its gradient and Hessian matrix?
I have been trying to use the scikit-learn library to solve this problem. Roughly:
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
# Make or ...
- 427
0
votes
1
answer
223
views
Python scipy.optimize.minimize: ‘trust-constr’ and Hessian output
I'm using scipy.optimize.minimize, method = 'trust-constr'.
Does anyone know how to retrieve the Hessian at the minimum when using method = ‘trust-constr’? I would like to use the Hessian to compute ...
- 11
1
vote
0
answers
148
views
Use pytorch to compute hessian of network, and the value is quite different from the theory
I'm currently training a two-layer neural network and want to compute the hessian w.r.t. the second-layer weights of the network. However, the output of the code is quite different from the theory. ...
- 11
0
votes
1
answer
113
views
autodiff for max()
I am doing optimization using maximum likelihood estimation, and when I am trying to get the standard errors of estimates using hessian matrix, I get non-invertible/singular hessian warning.
After I ...
- 263
0
votes
1
answer
643
views
How to efficiently compute the Hessian matrix of a Deep Neural Network?
I am using TF2.11.
In order to have a deeper understanding of PINNs, I want to compute the Hessian matrix of the loss wrt to my PINN parameters. My toy case is a 2D Poisson equation
$\Delta u = f$
I ...
- 124
2
votes
1
answer
597
views
Python Skimage: hessian() filter. Why are filter result values <= 0 set to 1?
I am currently working with ridge detection filters such as frangi(), sato() and hessian() within the python skimage package. In my project, I am using the hessian() filter to detect river-like ...
- 23
4
votes
2
answers
746
views
Detect wrinkle with Hessian matrix
Currently, I am trying to use Hessian matrix to detect wrinkles on the forehead. How could I remove out noises around these wrinkles? Below are my current code and result.
from skimage.feature import ...
- 144
1
vote
0
answers
171
views
How does scipy minimization handle cases where hessian is not positive definite
I am using from scipy.optimize import minimize to minimize a function subject to two constraints. I have been using the trust-constr method, which takes the value, gradient and the Hessian of the ...
- 327
0
votes
3
answers
235
views
Evaluate expression in environment passed to function as parameter in R
I am trying to create a function for theoretical hessian matrix that I can then evaluate at different locations. First I tried setting expressions as values in a matrix or array, but although I could ...
- 69
2
votes
1
answer
730
views
PyTorch: Compute Hessian matrix of the model
Say that, for some reason, I want to fit a linear regression using PyTorch, as illustrated below.
How could I compute the Hessian matrix of the model to, ultimately, compute the standard error for my ...
2
votes
1
answer
234
views
The output of functorch.hessian is not understandable
I am calculating the hessian of a ( 2,2) linear model using functorch.hessian as follows
model = torch.nn.Linear(2,2).to(device)
inputs = torch.rand(1,2).to(device)
criterion = torch.nn....
- 93
5
votes
1
answer
1k
views
Faster way to calculate the Hessian / Fisher Information Matrix of a nnet::multinom multinomial regression in R using Rcpp & Kronecker products
It appears that for larger nnet::multinom multinomial regression models (with a few thousand coefficients), calculating the Hessian (the matrix of second derivatives of the negative log likelihood, ...
- 8,069
0
votes
0
answers
725
views
UserWarning when using scipy.optimize.minimize with method 'trust-constr'
I am trying to minimize a function using scipy.optimize.minimize and I get the following errors:
Singular Jacobian matrix. Using SVD decomposition to perform the factorizations.
delta_grad == 0.0. ...
- 1,693
0
votes
1
answer
203
views
Necessity of Hessian Matrix
Hessian Matrix helps determine the saddle points, and the local extremum of a function (source)
Hessian Matrix is used in Newton methods to optimize functions. Of what use is a Hessian Matrix in ...
user16673741
1
vote
2
answers
870
views
Trying to understand the Hutchinson diagonal Hessian approximation
I am reading about his paper [1] and I have an implementation taken from here. At some point of the code the diagonal of the Hessian matrix is approximated by a function set_hessian you can find below....
- 922
2
votes
1
answer
708
views
Extract the diagonal elements of the Hessian in a neural network in Jax
I have a PyTree params (in my case a nested dictionary) containing my parameters of a neural network. My goal is to compute the diagonal entries of the Hessian of a loss function with respect to the ...
- 79
0
votes
1
answer
260
views
How do I find least square between two matrix in MATLAB?
What is the replacement in MATLAB for the following line of code snippet in python?
From Python Implementation for SIFT Feature Extraction
x = -lstsq(hessian, gradient, rcond=None)[0]
if
hessian = [-...
- 91
0
votes
0
answers
254
views
How to calculate Godambe information matrix in R?
I have a likelihood function in R that I am optimizing using 'optim' and calculating the hessian matrix using hessian=T in the optim function. I want to calculate the Godambe Information matrix in R, ...
0
votes
1
answer
1k
views
hessian matrix of a keras model with tf.hessians
I want to compute the Hessian matrix of a keras model w.r.t. its input in graph mode using tf.hessians.
Here is a minimal example
import tensorflow as tf
from tensorflow import keras
model = keras....
- 105
1
vote
0
answers
147
views
Hessian matrix w.r.t parameter of MLP is not symmetric (TF2.0)
I tried to calculate the Hessian matrix w.r.t. model parameters
However, each tensor shape in the Hessian matrix was not symmetric.
import tensorflow as tf
x_train = tf.constant(tf.random.uniform(...
- 77
1
vote
0
answers
405
views
How to properly specify Jacobian & Hessian functions of inequality constraints in Optim
I’m trying to use the Optim package in Julia to optimize an objective function with 19 variables, and the following inequality constraints:
0 <= x[1]/3 - x[2] <= 1/3
5 <= 1/x[3] + 1/x[4] <...
- 783
0
votes
1
answer
97
views
Does anybody understand how to make this code work?
I am trying to estimate a multiple linear probit model (killer_apps is the limited dependent variable) using the maximum likelihood approach. Therefore, in this code, I am trying to estimate the ...
2
votes
0
answers
2k
views
Hessian of Gaussian eigenvalues for 3D image with Python
I have a 3D image and I want to calculate the Hessian of Gaussian eigenvalues for this image. I would like to have the three eigenvalues of the Hessian approximation for each voxel. This feature seems ...
- 41
1
vote
1
answer
655
views
R: LTM: How can I make an odd-behaving hessian matrix converge when standardization fails?
I try to fit a graded response model with the R package ltm. The issue is that the Hessian matrix does not converge, and I don't understand why. Here is the code I use:
dset %>%
select(...
- 373
5
votes
1
answer
2k
views
How to implement a negative binomial loss function in python to use in light GBM?
I have a machine learning problem that I believe the negative binomial loss function would fit well, but the light gbm package doesn't have it as a standard, I'm trying to implement it, but I'm don't ...
- 123
2
votes
2
answers
2k
views
Cannot find in-place operation causing "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:"
I'm relatively new to PyTorch and am trying to reproduce an algorithm from an academic paper that approximates a term using the Hessian matrix. I've set up a toy problem so that I can compare the ...
- 23
0
votes
0
answers
1k
views
Why do I get negative variance from hessian matrix in optim function
I try to estimate mle parameters of a generalised gamma distribution.
I use optim function with a lower bound equal to one (since parameters must be positive) and BFGS method.
Initially, I estimate ...
- 1
-1
votes
1
answer
589
views
What is a Hessian matrix?
I know that the Hessian matrix is a kind of second derivative test of functions involving more than one independent variable. How does one find the maximum or minimum of a function involving more than ...
- 537
3
votes
1
answer
14k
views
Creating a Custom Objective Function in for XGBoost.XGBRegressor
So I am relatively new to the ML/AI game in python, and I'm currently working on a problem surrounding the implementation of a custom objective function for XGBoost.
My differential equation ...
- 31
4
votes
0
answers
384
views
How to compute the second derivatives (diagonal of the Hessian) in TensorFlow 2.0
I would like to compute the second derivatives (diagonal hessian) for all the components of all my variables in TensorFlow 2.0. I would like to autograph such a function as well.
I have it working ...
0
votes
0
answers
113
views
system throw exception when trying to invert hessian matrix
I'm trying some code from jamesmccaffrey web about inverse matrix (link). I used for invert n x n hessian matrix and I got the exception: "Cannot use Doolittle's method" in "static double[][] ...
0
votes
0
answers
1k
views
Sympy: Calculating eigenvalues takes very long for (9,9) matrix
I'm trying to calculate the eigenvalues of a matrix F, which contains 9 variables, which are the cartesian coordinates of three vectors. The execution time for F.eigenvals() takes at least 15 minutes ...
- 534
0
votes
1
answer
723
views
Sympy: Solve entries in Hessian Matrix for better readability?
I'm very green when it comes to sympy and I don't know how to produce output in a well formatted fashion. Right now I've computed the Hessian matrix of my potential function:
V = 1/2*kOH*(r1)**2 +1/2*...
- 534
6
votes
1
answer
6k
views
defining the Hessian as zero
While using scipy.optimize.minimize with the trust-constr method I got this UserWarning:
scipy\optimize\_hessian_update_strategy.py:187: UserWarning: delta_grad == 0.0. Check if the approximated ...
- 327
2
votes
0
answers
794
views
Why do the standard errors obtained via lsqcurvefit and fminunc function differ?
I am fitting a model to experimental data in Matlab. For this model I wanted to find the parameters by minimising the sum of square residues between the experimental and model data set. The model is ...
3
votes
1
answer
790
views
Hessian matrix, how to combine Ixx & Iyy together?
"Before extracting the lines, you need to detect potential points on them. Apply a Gaussian filter first and use the Sobel filters as derivative operators. Threshold the determinant of the Hessian and ...
- 31
2
votes
0
answers
815
views
How to obtain second derivatives of a Loss function with respect to the parameters of a neural network using gradient tape in Tensorflow eager mode
I am creating a basic auto-encoder for the MNIST dataset using TensorFlow eager mode. I would like to observe the second-order partial derivatives of my loss function with respect to the parameters of ...
- 118
1
vote
1
answer
472
views
How can I get standard errors for my 4 parameters when the Hessian matrix from solnp is 5 by 5?
I'm using the solnp() function in the R package Rsolnp to solve a nonlinear regression with constraints. It works well, converges with no problem. I want to use the Hessian matrix to calculate ...
- 4,804
3
votes
1
answer
1k
views
Compute hessian with respect to several variables in tensorflow
Computing Hessian in tensorflow is quite easy:
x = tf.Variable([1., 1., 1.], dtype=tf.float32, name="x")
f = (x[0] + x[1] ** 2 + x[0] * x[1] + x[2]) ** 2
hessian = tf.hessians(f, x)
This correctly ...
- 17.8k
1
vote
1
answer
329
views
hessian for inequality constraint in fmincon
I am trying to help fmincon to converge faster by supplying gradient vector and Hessian matrix. I am using interior-point algorithm and I realize that in such case, I have to supply Hessian using a ...
- 19
3
votes
1
answer
273
views
Reshaping numpy array
What I am trying to do is take a numpy array representing 3D image data and calculate the hessian matrix for every voxel. My input is a matrix of shape (Z,X,Y) and I can easily take a slice along z ...
- 154
1
vote
1
answer
436
views
Minimize function won't show me the hessian
I minimized a function and need it's (inverse) Hessian for the standard errors.
the function gives me this for the (inverse) Hessian:
hess_inv: <5x5 LbfgsInvHessProduct with dtype=float64>
I ...
- 187
4
votes
1
answer
2k
views
Use tf.gradients or tf.hessians on flattened parameter tensor
Let's say I want to compute the Hessian of a scalar-valued function with respect to some parameters W (e.g the weights and biases of a feed-forward neural network).
If you consider the following code, ...
- 98
3
votes
1
answer
3k
views
XGBoost python custom objective function
According to documentation,
http://xgboost.readthedocs.io/en/latest/python/python_api.html
if we want to define custom objective function, it should have signature
objective(y_true, y_pred) -> ...
- 31
2
votes
1
answer
531
views
Using TensorFlow hessians for second partial derivative test
Second partial derivative test is a simple way to tell whether a critical point is a minimum, a maximum, or a saddle. I am currently toying with the idea of implementing such test for a simple neural ...
- 189
0
votes
1
answer
92
views
TensorFlow Hessian matrix is not updated after training session
I am trying to get Hessian matrix using tf.hessians function. Whereas the loss value and variables are updated after each training session, Hessian matrix values remain constant. Moreover, they does ...
3
votes
1
answer
842
views
Retrieve optimization results from MLE by scipy.stats.fit()?
I am trying to estimate different distributions' parameters via scipy.stats."some_dist".fit(), and I'm having extreme difficulty retrieving any information on the optimization process being used. ...
- 1,769
2
votes
1
answer
1k
views
Substitute values for variables in the result of derive_by_array in SymPy
I'm currently working through some exercises on multivariable function calculus and thought I would have a go at making my own function to determine gradient and hessian at a defined point for any ...
- 1,841
3
votes
0
answers
793
views
Hessian matrix with optim or numderiv package?
I do maximum likelihood estimation for a loglikelihood function of a poisson distribution. After the estimation i want to compute the standard errors of the coeffients. For that i need the hessian ...
- 243