I can have some interest in, hope for, etc. machine learning. One reason is, for the curve fitting methods of classic statistics, i.e., versions of regression, the math assumptions that give some hope of some good results are essentially impossible to verify and look like they will hold closely only rarely. So, even when using such statistics, good advice is to have two steps, (1) apply the statistics, i.e., fit, using half the data and then (2) verify, test, check using the other half. But, gee, those two steps are also common in machine learning. Sooo, if can't find much in classic math theorems and proofs to support machine learning, then, are just put back into the two steps statistics has had to use anyway.
So, if we have to use the two steps anyway, then the possible advantages of non-linear fitting have some promise.
So, to me, a larger concern comes to the top: In my experience in such things, call it statistics, optimization, data analysis, whatever, a huge advantage is bringing to the work some understanding that doesn't come with the data and/or really needs a human. The understanding might be about the real problem or about some mathematical methods.
E.g., once some guys had a problem in optimal allocation of some resources. They had tried simulated annealing, run for days, and quit without knowing much about the quality of the results.
I took the problem as 0-1 integer linear programming, a bit large, 600,000 variables, 40,000 constraints, and in 900 seconds on a slow computer, with Lagrangian relaxation, got a feasible solution guaranteed, from the bounding, to be within 0.025% of optimality. The big advantages were understanding the 0-1 program, seeing a fast way to do the primal-dual iterations, and seeing how to use Lagrangian relaxation. My guess is that it would be tough for some very general machine learning to compete much short of artificial general intelligence.
One way to describe the problem with the simulated annealing was that it was just too general, didn't exploit what a human might understand about the real problem and possible solution methods selected for that real problem.
I have a nice collection of such successes where the keys were some insight into the specific problems and some math techniques, that is, some human abilities that would seem to need machine learning to have artificial general intelligence to compete. With lots of data, lots of computing, and the advantages of non-linear operations, at times machine learning might be the best approach even now.
Net, still, in many cases, human intelligence is tough to beat.
A point about gradient-free methods such as simulated annealing and genetic algorithms: the transition (sometimes called "neighbor") function is the most important part by far. The most important insight is the most obvious one in some way: if your task is to search a problem space efficiently for an optimal solution, it pays to know exactly how to move from where you are to where you want to be in that problem space. To that point, (the structure of) transitions between successive state samples should be refined to your specific problem and encoding of the domain in order to be useful in any reasonable amount of time.
> the transition (sometimes called "neighbor") function is the most important part by far.
And, indeed, in the 0-1 integer linear programming with Lagrangian relaxation I used there is nothing differentiable so should be counted as "gradient free". And the linear programming part and the Lagrangian part do "move" from where are to closer to "where want to be".
A thing is, the bag of tricks, techniques, that work here is large. So, right, should use knowledge of the real problem to pick what tricks to use.
So, if we have to use the two steps anyway, then the possible advantages of non-linear fitting have some promise.
So, to me, a larger concern comes to the top: In my experience in such things, call it statistics, optimization, data analysis, whatever, a huge advantage is bringing to the work some understanding that doesn't come with the data and/or really needs a human. The understanding might be about the real problem or about some mathematical methods.
E.g., once some guys had a problem in optimal allocation of some resources. They had tried simulated annealing, run for days, and quit without knowing much about the quality of the results.
I took the problem as 0-1 integer linear programming, a bit large, 600,000 variables, 40,000 constraints, and in 900 seconds on a slow computer, with Lagrangian relaxation, got a feasible solution guaranteed, from the bounding, to be within 0.025% of optimality. The big advantages were understanding the 0-1 program, seeing a fast way to do the primal-dual iterations, and seeing how to use Lagrangian relaxation. My guess is that it would be tough for some very general machine learning to compete much short of artificial general intelligence.
One way to describe the problem with the simulated annealing was that it was just too general, didn't exploit what a human might understand about the real problem and possible solution methods selected for that real problem.
I have a nice collection of such successes where the keys were some insight into the specific problems and some math techniques, that is, some human abilities that would seem to need machine learning to have artificial general intelligence to compete. With lots of data, lots of computing, and the advantages of non-linear operations, at times machine learning might be the best approach even now.
Net, still, in many cases, human intelligence is tough to beat.