Discussion:
SciPy improvements
(too old to reply)
Ondrej Certik
2007-04-11 17:02:42 UTC
Permalink
Hi,

I am studying theoretical physics and I have collected a lot of useful
python code, that I believe could go to SciPy. So first I'll review
what I have and if you find it interesting, I would like to discuss
how I could implement it in SciPy.


1) Optimization

http://chemev.googlecode.com/svn/trunk/chemev/optimization/

I have a Differential Evolution optimizer, Simplex optimizer, mcmc
(not well tested yet), I took a code from someone else, but adapted
the interface to the SciPy's one:

def fmin_de(f,x0,callback=None,iter=None):

Those are unconstrained optimizers. Then I have a constrains code,
that applies a logistic function to the fitting variable and allows me
to do constrained optimization. For example the L-BFGS with my
constrains converges 7x faster, than the original L-BFGS-B on my
problem.

2) Nonlinear solvers

I have written these nonlinear solvers for the problem R(x) = 0, where
x and R has a dimension "n":

broyden1 - Broyden's first method - is a quasi-Newton-Raphson method for
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates the
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead of
directly computing the inverse Jacobian, it remembers how to construct
it using vectors, and when computing inv(J)*F, it uses those vectors to
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as broyden2,
but instead of approximating the full NxN Jacobian, it construct it at
every iteration in a way that avoids the NxN matrix multiplication.
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the broyden_generalized,
but added w_0^2*I to before taking inversion to improve the stability
anderson2 - the Anderson method, the same as anderson, but formulated
differently
linear_mixing
exciting_mixing

I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).

Also I am writing a BFGS solver with linesearch, that should behave
even better than the Broyden scheme.

Of course I am trying to use SciPy's code (like linesearch) wherever possible.

3) PETSC bindings

I found these nice petsc bindings:

http://cheeseshop.python.org/pypi/petsc4py/0.7.2

I believe this could also be an optional package in SciPy. Because if
SciPy has some sparse matrices code, then it should definitely also
has this.

4) Finite element code?

I have my own code, that uses libmesh:

http://libmesh.sourceforge.net/

and calls tetgen and parses input from gmsh etc. Can convert the mesh,
can refine it, can solve it, etc. Webpages are here:

http://code.google.com/p/femgeom/
http://code.google.com/p/libmeshpetscpackage/
http://code.google.com/p/grainmodel/

I am not sure here, if it should belong to SciPy. Probably not.

5) Symbolic manipulation in Python

http://code.google.com/p/sympy/

We'll have some google summer of code students working on this and
also I am not sure if it belongs to SciPy. However, this project looks
very promising.


-----------------
So that's it.

I have some comments to SciPy:

1) Documentation

Virtually none, I just use the source code to understand, what SciPy
can do and how. But the docstrings are good though. I would suggest to
update the

http://www.scipy.org/doc/api_docs/

more often (for example I didn't find there the new l-bfgs code).

2) What is the official NumPy's page?

I believe it should be

http://numpy.org/

however, it points to a sourceforge page. I believe this homepage
should contain all the relevant information about numpy and contain
links to the fee based documentation and possibly some tutorials.
The documentation of NumPy is scattered across many pages and I find
it confusing.

I know you have some list here:

http://www.scipy.org/MigratingFromPlone

But I am quite confused from the whole SciPy page. I think less but
unconfusing information is better, but that's just my opinion. I think
the front page of both SciPy and NumPy should be clean and simple with
a clear link to a documentation. Like

http://www.pytables.org/moin/PyTables
http://matplotlib.sourceforge.net/

However, the documentation:

http://www.scipy.org/Documentation

is confusing, because except the fee based guide to NumPy, it's not
clear, what is the official SciPy doc and what the best way of
learning SciPy is.

So I am interested in your opinions and then I'll just integrate my
code into scipy.optimize and send you a patch to this list?

Ondrej
Robert Kern
2007-04-11 20:09:13 UTC
Permalink
Post by Ondrej Certik
Hi,
I am studying theoretical physics and I have collected a lot of useful
python code, that I believe could go to SciPy. So first I'll review
what I have and if you find it interesting, I would like to discuss
how I could implement it in SciPy.
Excellent! Thank you! One thing to be aware of is that scipy uses the BSD
license, so you would need to relicense your code under the BSD license and get
permission from the others who have contributed to the code you are submitting.
Post by Ondrej Certik
1) Optimization
http://chemev.googlecode.com/svn/trunk/chemev/optimization/
I have a Differential Evolution optimizer, Simplex optimizer, mcmc
(not well tested yet), I took a code from someone else, but adapted
Well, fmin() is already an implementation of the simplex algorithm. How does
yours compare? We can't include the MCMC optimizer until we have an
implementation of Metropolis-Hastings in scipy itself; we're not going to depend
on an external PyMC. As for the differential evolution code, with all respect to
you and Jame Phillips, it's not the best way to implement that algorithm in
Python. It's a straight translation of the C++ code so it doesn't make use of
numpy at all. I have an implementation that does:

http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/rkern/diffev.py

It was written for pre-numpy scipy, so it may need some sprucing-up before it works.
Post by Ondrej Certik
Those are unconstrained optimizers. Then I have a constrains code,
that applies a logistic function to the fitting variable and allows me
to do constrained optimization. For example the L-BFGS with my
constrains converges 7x faster, than the original L-BFGS-B on my
problem.
Interesting. Let's toss it up on the Cookbook first and pound on it a bit. I
have qualms about applying such transformations to the domains of target
functions and then using derivative-based optimizers on them, but those qualms
might be baseless. Still, I'd rather experiment first before putting them into
scipy.
Post by Ondrej Certik
2) Nonlinear solvers
I have written these nonlinear solvers for the problem R(x) = 0, where
broyden1 - Broyden's first method - is a quasi-Newton-Raphson method for
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates the
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead of
directly computing the inverse Jacobian, it remembers how to construct
it using vectors, and when computing inv(J)*F, it uses those vectors to
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as broyden2,
but instead of approximating the full NxN Jacobian, it construct it at
every iteration in a way that avoids the NxN matrix multiplication.
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the broyden_generalized,
but added w_0^2*I to before taking inversion to improve the stability
anderson2 - the Anderson method, the same as anderson, but formulated
differently
linear_mixing
exciting_mixing
I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).
Also I am writing a BFGS solver with linesearch, that should behave
even better than the Broyden scheme.
Of course I am trying to use SciPy's code (like linesearch) wherever possible.
That's fantastic. I'd love to see them. Are they in chemev? I don't see them.
Post by Ondrej Certik
3) PETSC bindings
http://cheeseshop.python.org/pypi/petsc4py/0.7.2
I believe this could also be an optional package in SciPy. Because if
SciPy has some sparse matrices code, then it should definitely also
has this.
I don't know. It has a fine (and probably better) existence separate from scipy.
Post by Ondrej Certik
4) Finite element code?
http://libmesh.sourceforge.net/
and calls tetgen and parses input from gmsh etc. Can convert the mesh,
http://code.google.com/p/femgeom/
http://code.google.com/p/libmeshpetscpackage/
http://code.google.com/p/grainmodel/
I am not sure here, if it should belong to SciPy. Probably not.
I think you are right. You can't really get around the licenses of your
dependencies, here.
Post by Ondrej Certik
5) Symbolic manipulation in Python
http://code.google.com/p/sympy/
We'll have some google summer of code students working on this and
also I am not sure if it belongs to SciPy. However, this project looks
very promising.
Again, I think it has a fine existence separate from scipy. A reason to bring it
into scipy would be such that other parts of scipy would use it to implement
their own stuff. Otherwise, I don't think there is much point.
Post by Ondrej Certik
-----------------
So that's it.
1) Documentation
Virtually none, I just use the source code to understand, what SciPy
can do and how. But the docstrings are good though. I would suggest to
update the
http://www.scipy.org/doc/api_docs/
more often (for example I didn't find there the new l-bfgs code).
2) What is the official NumPy's page?
http://numpy.scipy.org
Post by Ondrej Certik
I believe it should be
http://numpy.org/
however, it points to a sourceforge page.
Correct. I don't know who still owns that domain.
Post by Ondrej Certik
I believe this homepage
should contain all the relevant information about numpy and contain
links to the fee based documentation and possibly some tutorials.
The documentation of NumPy is scattered across many pages and I find
it confusing.
http://www.scipy.org/MigratingFromPlone
But I am quite confused from the whole SciPy page. I think less but
unconfusing information is better, but that's just my opinion. I think
the front page of both SciPy and NumPy should be clean and simple with
a clear link to a documentation. Like
http://www.pytables.org/moin/PyTables
http://matplotlib.sourceforge.net/
Please, by all means submit your recommendations for reorganization of that
page. Because the front page is special, I'd recommend submitting your
modifications as a ticket to our Trac (see below) instead of just editing it.
The other pages on the wiki, please modify them as you see fit.
Post by Ondrej Certik
http://www.scipy.org/Documentation
is confusing, because except the fee based guide to NumPy, it's not
clear, what is the official SciPy doc and what the best way of
learning SciPy is.
There really is no official scipy doc at this time. That's part of the problem.
Post by Ondrej Certik
So I am interested in your opinions and then I'll just integrate my
code into scipy.optimize and send you a patch to this list?
Register an account with the scipy Trac (click "Register" in the upper-right
corner):

http://projects.scipy.org/scipy/scipy

Then make a new ticket and attach your patch to that. Submit enough patches, and
we'll just give you SVN access.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Ondrej Certik
2007-04-24 13:20:50 UTC
Permalink
Post by Robert Kern
Register an account with the scipy Trac (click "Register" in the upper-right
http://projects.scipy.org/scipy/scipy
Then make a new ticket and attach your patch to that. Submit enough patches, and
we'll just give you SVN access.
Hi, the patch is here:

http://projects.scipy.org/scipy/scipy/ticket/402

it should be enough to apply it in the scipy root dir.

Notes:
I created a new module nonlin, and put the solvers and tests (I
adapted them to the scipy test framework). I am interested in
criticisms, like if I should better put it into the optimize module (I
think optimization is a different field), or what else should be done.

I have a question about how you work when developing scipy? I do:

1) I play in the trunk, implement something
2) in the scipy root dir I execute:

rm -rf ../dist/; ./setup.py install --prefix ../dist

3) in the parent directory, I have execute this short script, that
tests, that my change in 1) works fine:

import sys
sys.path.insert(0,"dist/lib/python2.4/site-packages/")
import scipy
if scipy.version.release:
raise "The svn version not imported!! Fix your paths"

from scipy import nonlin

nonlin.test()


However, I am interested, if you have some better approach.

Thanks,
Ondrej

Matthieu Brucher
2007-04-11 20:56:45 UTC
Permalink
Post by Ondrej Certik
2) Nonlinear solvers
I have written these nonlinear solvers for the problem R(x) = 0, where
broyden1 - Broyden's first method - is a quasi-Newton-Raphson method for
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates the
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead of
directly computing the inverse Jacobian, it remembers how to construct
it using vectors, and when computing inv(J)*F, it uses those vectors to
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as broyden2,
but instead of approximating the full NxN Jacobian, it construct it at
every iteration in a way that avoids the NxN matrix
multiplication.
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the
broyden_generalized,
but added w_0^2*I to before taking inversion to improve the stability
anderson2 - the Anderson method, the same as anderson, but formulated
differently
linear_mixing
exciting_mixing
I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).
Could the part that computes the step be separated from the function iself
and the optimizer ? I'm trying to "modularize" non linear solvers so as to
select more efficiently what is needed - kind of optimizer, kind of step,
kind of stopping criterion, ... -

Matthieu
Michael McNeil Forbes
2007-04-12 21:31:54 UTC
Permalink
Post by Ondrej Certik
2) Nonlinear solvers
I have written these nonlinear solvers for the problem R(x) = 0, where
broyden1 - Broyden's first method - is a quasi-Newton-Raphson method for
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates the
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead of
directly computing the inverse Jacobian, it remembers how to construct
it using vectors, and when computing inv(J)*F, it uses
those vectors to
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as broyden2,
but instead of approximating the full NxN Jacobian, it
construct it at
every iteration in a way that avoids the NxN matrix
multiplication.
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the
broyden_generalized,
but added w_0^2*I to before taking inversion to improve the stability
anderson2 - the Anderson method, the same as anderson, but
formulated
differently
linear_mixing
exciting_mixing
I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).
Could the part that computes the step be separated from the
function iself and the optimizer ? I'm trying to "modularize" non
linear solvers so as to select more efficiently what is needed -
kind of optimizer, kind of step, kind of stopping criterion, ... -
Matthieu
It should be possible to modularize these with a step class that
maintains state
(the Jacobian, or its inverse etc.).

(Where is the latest version of your optimization proposal? I have
not had a chance
to look at it yet, but have been meaning to and would like to look at
the latest version.
Perhaps we should make a page on the Wiki to collect suggestions and
code samples.)

I have been meaning to get a good Broyden based algorithm coded for
python for a
while. I have a MATLAB version of a globally convergent Broyden
implementation
using a linesearch as a base with a couple of unusual features that
might be
useful (not specific to Broyden based methods). (The code is based
on the
presentation of the algorithm given in Numerical Recipies with some
modifications
suggested by Dennis and Schnabel's book and is partially documented
using noweb.)

http://www.phys.washington.edu/users/mforbes/projects/broyden/

1) Variable tolerances. The idea is to quickly estimate the starting
Jacobian
with low tolerance calculations and then improve the tolerances
as the code
converges to the solution. This is useful if the function R(x)
is computed
with numerical integration or some similar technique that is
quick for low
tolerances but expensive for high tolerance functions. (I have
rarely seen
this technique in generic optimization algorithms, but found it
invaluable for
several problems.)
2) Real-time bailout. This allows you to compute for a specified
length of time
and then return the best solution found within that time frame.
Most algorithms
simply count function calls.
3) Selective refinement of the Jacobian as the iteration proceeds.
This amounts
to monitoring the condition number of the Jacobian and
recomputing parts of it
selectively if it becomes ill-conditioned. (For this I update
the QR
factorization of J rather than maintaining inv(J)).

These things all slow down the fundamental algorithm, but are very
useful when the
function R(x) is extremely expensive to evaluate.

Michael.
Ondrej Certik
2007-04-12 23:31:35 UTC
Permalink
Hi,

thank to all of you for your thorough response.

1) The DE code from Robert:

http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/rkern/diffev.py

looks incredibly simple, I'll check it on my problem, if it behaves
better than my ugly one. But that's the thing - I didn't find any
mention about it in the documentation, otherwise I would be using your
code, because it's simpler (=better). Also I didn't find it in google.

I'll check the simplex method from SciPy - when I was using it, I just
needed a simple script, not a whole dependence on SciPy and it was
very difficult to get just the simplex algorithm out of SciPy.

2) I am sending my Broyden update methods in the attachement together
with tests (you need py.test to run them). I am using a test from some
article, I think from Vanderbilt. However my code is giving a little
different results. I'll polish it and send it as a patch to SciPy in
the trac, as directed, so you can just look how I do it. But when you
do it, you will see there is really nothing to it - it's very trivial.

However, my code doesn't use linesearch. Also I am curious how the
BFGS method is going to work when I implement it - it's very similar
to Broyden, except the update of the Jacobian is a little different.

Could you Michael please also rewrite your code to Python?

http://www.phys.washington.edu/users/mforbes/projects/broyden/

It would be nice to have all of it in SciPy with the same interface.

BTW - why are you using matlab at all? To me, the python with numpy is
better than anything else I've programmed in, including matlab.

3) about the logistics transforamtion - I was sceptical too, until I
tried that and it was converging faster by a factor of 7x on my
problem (chemev). So for me it's enough justification, but of course I
am not saying that it must converge faster for any problem.

However, I implemented it as a wrapper function above all the
unconstrained algorithms with the SciPy interface, so the user is not
forced to use it - he can just try it and see, as I did. I'll post it
to the cookbook.

4) About the petsc - I know it's another dependence. However, I
noticed you are using umfpack in SciPy. So why not petsc? I think it
contains much more (sometimes better) solvers (depends on the
problem). It's seems logical to me, to either use nothing, or the best
library available, which I believe is petsc.

5)documentation: the front page is quite fine, however the
documentation needs complete redesign in my opinion. First - I believe
the numpy should be separated from SciPy and have it's own page
(numpy.org), but if you think it belongs under the hood of scipy.org,
then ok.

So, I'll copy the page:

http://www.scipy.org/Documentation

into some new one, and redesign it as I would like it to be, and then
you'll tell me what you think about it. The same with other pages if
I'll get a better idea about them. This way I shouldn't spoil anything
in case you wouldn't like it. Because I don't have just couple of
small fixes.

Ondrej
Post by Ondrej Certik
Post by Ondrej Certik
2) Nonlinear solvers
I have written these nonlinear solvers for the problem R(x) = 0, where
broyden1 - Broyden's first method - is a quasi-Newton-Raphson method
for
Post by Ondrej Certik
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates
the
Post by Ondrej Certik
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead
of
Post by Ondrej Certik
directly computing the inverse Jacobian, it remembers how to
construct
Post by Ondrej Certik
it using vectors, and when computing inv(J)*F, it uses those
vectors to
Post by Ondrej Certik
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as
broyden2,
Post by Ondrej Certik
but instead of approximating the full NxN Jacobian, it construct
it at
Post by Ondrej Certik
every iteration in a way that avoids the NxN matrix
multiplication.
Post by Ondrej Certik
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the
broyden_generalized,
Post by Ondrej Certik
but added w_0^2*I to before taking inversion to improve the
stability
Post by Ondrej Certik
anderson2 - the Anderson method, the same as anderson, but formulated
differently
linear_mixing
exciting_mixing
I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).
Could the part that computes the step be separated from the function iself
and the optimizer ? I'm trying to "modularize" non linear solvers so as to
select more efficiently what is needed - kind of optimizer, kind of step,
kind of stopping criterion, ... -
Matthieu
It should be possible to modularize these with a step class that maintains
state
(the Jacobian, or its inverse etc.).
(Where is the latest version of your optimization proposal? I have not had
a chance
to look at it yet, but have been meaning to and would like to look at the
latest version.
Perhaps we should make a page on the Wiki to collect suggestions and code
samples.)
I have been meaning to get a good Broyden based algorithm coded for python
for a
while. I have a MATLAB version of a globally convergent Broyden
implementation
using a linesearch as a base with a couple of unusual features that might be
useful (not specific to Broyden based methods). (The code is based on the
presentation of the algorithm given in Numerical Recipies with some
modifications
suggested by Dennis and Schnabel's book and is partially documented using
noweb.)
http://www.phys.washington.edu/users/mforbes/projects/broyden/
1) Variable tolerances. The idea is to quickly estimate the starting
Jacobian
with low tolerance calculations and then improve the tolerances as the
code
converges to the solution. This is useful if the function R(x) is
computed
with numerical integration or some similar technique that is quick for
low
tolerances but expensive for high tolerance functions. (I have rarely
seen
this technique in generic optimization algorithms, but found it
invaluable for
several problems.)
2) Real-time bailout. This allows you to compute for a specified length of
time
and then return the best solution found within that time frame. Most
algorithms
simply count function calls.
3) Selective refinement of the Jacobian as the iteration proceeds. This
amounts
to monitoring the condition number of the Jacobian and recomputing parts
of it
selectively if it becomes ill-conditioned. (For this I update the QR
factorization of J rather than maintaining inv(J)).
These things all slow down the fundamental algorithm, but are very useful
when the
function R(x) is extremely expensive to evaluate.
Michael.
_______________________________________________
Scipy-dev mailing list
http://projects.scipy.org/mailman/listinfo/scipy-dev
Robert Kern
2007-04-12 23:45:15 UTC
Permalink
Post by Ondrej Certik
3) about the logistics transforamtion - I was sceptical too, until I
tried that and it was converging faster by a factor of 7x on my
problem (chemev). So for me it's enough justification, but of course I
am not saying that it must converge faster for any problem.
I'm sure it works faster; I'd just like to make sure that it always gives the
correct answer.
Post by Ondrej Certik
4) About the petsc - I know it's another dependence. However, I
noticed you are using umfpack in SciPy. So why not petsc? I think it
contains much more (sometimes better) solvers (depends on the
problem). It's seems logical to me, to either use nothing, or the best
library available, which I believe is petsc.
Well, I wasn't as much of a dependency/no-optional-features freak when the
optional UMFPACK stuff went in. Also, IIRC the wrappers for UMFPACK were written
specifically for scipy; they didn't exist as a separate package beforehand.
petsc4py already exists. Unless if we decide that some other feature of scipy
needs it, there is no reason that I can see for bringing it into the scipy package.
Post by Ondrej Certik
5)documentation: the front page is quite fine, however the
documentation needs complete redesign in my opinion. First - I believe
the numpy should be separated from SciPy and have it's own page
(numpy.org), but if you think it belongs under the hood of scipy.org,
then ok.
http://www.scipy.org/Documentation
into some new one, and redesign it as I would like it to be, and then
you'll tell me what you think about it. The same with other pages if
I'll get a better idea about them. This way I shouldn't spoil anything
in case you wouldn't like it. Because I don't have just couple of
small fixes.
As you like. Thank you!
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Robert Cimrman
2007-04-13 10:59:33 UTC
Permalink
Post by Robert Kern
Post by Ondrej Certik
4) About the petsc - I know it's another dependence. However, I
noticed you are using umfpack in SciPy. So why not petsc? I think it
contains much more (sometimes better) solvers (depends on the
problem). It's seems logical to me, to either use nothing, or the best
library available, which I believe is petsc.
Well, I wasn't as much of a dependency/no-optional-features freak when the
optional UMFPACK stuff went in. Also, IIRC the wrappers for UMFPACK were written
specifically for scipy; they didn't exist as a separate package beforehand.
petsc4py already exists. Unless if we decide that some other feature of scipy
needs it, there is no reason that I can see for bringing it into the scipy package.
Yes, it was written specifically for scipy. Actually I was really
'forced' to write UMFPACK wrappers as at that time (long long ago) there
was not a fast enough direct sparse solver in scipy.

r.
Ondrej Certik
2007-04-13 23:57:53 UTC
Permalink
Post by Robert Kern
Post by Ondrej Certik
http://www.scipy.org/Documentation
into some new one, and redesign it as I would like it to be, and then
you'll tell me what you think about it. The same with other pages if
I'll get a better idea about them. This way I shouldn't spoil anything
in case you wouldn't like it. Because I don't have just couple of
small fixes.
As you like. Thank you!
You can check my draft of a new documentation:

http://www.scipy.org/DocumentationNew

and the original one:

http://www.scipy.org/Documentation

I removed broken links, merged very simple pages with the tutorial and
removed duplicates. Tell me, if you like it and if I should continue -
I would merge SciPy Tutorial with Tutorial II, then I would merge all
porting wikis into one central with links (or just add links to one of
them) and possibly some more simplifying - it's still too much
complicated with too much (confusing) links.

Ondrej
Ondrej Certik
2007-04-12 23:32:14 UTC
Permalink
Hi,

thank to all of you for your thorough response.

1) The DE code from Robert:

http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/rkern/diffev.py

looks incredibly simple, I'll check it on my problem, if it behaves
better than my ugly one. But that's the thing - I didn't find any
mention about it in the documentation, otherwise I would be using your
code, because it's simpler (=better). Also I didn't find it in google.

I'll check the simplex method from SciPy - when I was using it, I just
needed a simple script, not a whole dependence on SciPy and it was
very difficult to get just the simplex algorithm out of SciPy.

2) I am sending my Broyden update methods in the attachement together
with tests (you need py.test to run them). I am using a test from some
article, I think from Vanderbilt. However my code is giving a little
different results. I'll polish it and send it as a patch to SciPy in
the trac, as directed, so you can just look how I do it. But when you
do it, you will see there is really nothing to it - it's very trivial.

However, my code doesn't use linesearch. Also I am curious how the
BFGS method is going to work when I implement it - it's very similar
to Broyden, except the update of the Jacobian is a little different.

Could you Michael please also rewrite your code to Python?

http://www.phys.washington.edu/users/mforbes/projects/broyden/

It would be nice to have all of it in SciPy with the same interface.

BTW - why are you using matlab at all? To me, the python with numpy is
better than anything else I've programmed in, including matlab.

3) about the logistics transforamtion - I was sceptical too, until I
tried that and it was converging faster by a factor of 7x on my
problem (chemev). So for me it's enough justification, but of course I
am not saying that it must converge faster for any problem.

However, I implemented it as a wrapper function above all the
unconstrained algorithms with the SciPy interface, so the user is not
forced to use it - he can just try it and see, as I did. I'll post it
to the cookbook.

4) About the petsc - I know it's another dependence. However, I
noticed you are using umfpack in SciPy. So why not petsc? I think it
contains much more (sometimes better) solvers (depends on the
problem). It's seems logical to me, to either use nothing, or the best
library available, which I believe is petsc.

5)documentation: the front page is quite fine, however the
documentation needs complete redesign in my opinion. First - I believe
the numpy should be separated from SciPy and have it's own page
(numpy.org), but if you think it belongs under the hood of scipy.org,
then ok.

So, I'll copy the page:

http://www.scipy.org/Documentation

into some new one, and redesign it as I would like it to be, and then
you'll tell me what you think about it. The same with other pages if
I'll get a better idea about them. This way I shouldn't spoil anything
in case you wouldn't like it. Because I don't have just couple of
small fixes.

Ondrej
Post by Ondrej Certik
Post by Ondrej Certik
2) Nonlinear solvers
I have written these nonlinear solvers for the problem R(x) = 0, where
broyden1 - Broyden's first method - is a quasi-Newton-Raphson method
for
Post by Ondrej Certik
updating an approximate Jacobian and then inverting it
broyden2 - Broyden's second method - the same as broyden1, but updates
the
Post by Ondrej Certik
inverse Jacobian directly
broyden3 - Broyden's second method - the same as broyden2, but instead
of
Post by Ondrej Certik
directly computing the inverse Jacobian, it remembers how to
construct
Post by Ondrej Certik
it using vectors, and when computing inv(J)*F, it uses those
vectors to
Post by Ondrej Certik
compute this product, thus avoding the expensive NxN matrix
multiplication.
broyden_generalized - Generalized Broyden's method, the same as
broyden2,
Post by Ondrej Certik
but instead of approximating the full NxN Jacobian, it construct
it at
Post by Ondrej Certik
every iteration in a way that avoids the NxN matrix
multiplication.
Post by Ondrej Certik
This is not as precise as broyden3.
anderson - extended Anderson method, the same as the
broyden_generalized,
Post by Ondrej Certik
but added w_0^2*I to before taking inversion to improve the
stability
Post by Ondrej Certik
anderson2 - the Anderson method, the same as anderson, but formulated
differently
linear_mixing
exciting_mixing
I use them in the self-consistent cycle of the Density Functional
Theory (so I use a terminology of DFT literature in the names of the
methods).
Could the part that computes the step be separated from the function iself
and the optimizer ? I'm trying to "modularize" non linear solvers so as to
select more efficiently what is needed - kind of optimizer, kind of step,
kind of stopping criterion, ... -
Matthieu
It should be possible to modularize these with a step class that maintains
state
(the Jacobian, or its inverse etc.).
(Where is the latest version of your optimization proposal? I have not had
a chance
to look at it yet, but have been meaning to and would like to look at the
latest version.
Perhaps we should make a page on the Wiki to collect suggestions and code
samples.)
I have been meaning to get a good Broyden based algorithm coded for python
for a
while. I have a MATLAB version of a globally convergent Broyden
implementation
using a linesearch as a base with a couple of unusual features that might be
useful (not specific to Broyden based methods). (The code is based on the
presentation of the algorithm given in Numerical Recipies with some
modifications
suggested by Dennis and Schnabel's book and is partially documented using
noweb.)
http://www.phys.washington.edu/users/mforbes/projects/broyden/
1) Variable tolerances. The idea is to quickly estimate the starting
Jacobian
with low tolerance calculations and then improve the tolerances as the
code
converges to the solution. This is useful if the function R(x) is
computed
with numerical integration or some similar technique that is quick for
low
tolerances but expensive for high tolerance functions. (I have rarely
seen
this technique in generic optimization algorithms, but found it
invaluable for
several problems.)
2) Real-time bailout. This allows you to compute for a specified length of
time
and then return the best solution found within that time frame. Most
algorithms
simply count function calls.
3) Selective refinement of the Jacobian as the iteration proceeds. This
amounts
to monitoring the condition number of the Jacobian and recomputing parts
of it
selectively if it becomes ill-conditioned. (For this I update the QR
factorization of J rather than maintaining inv(J)).
These things all slow down the fundamental algorithm, but are very useful
when the
function R(x) is extremely expensive to evaluate.
Michael.
_______________________________________________
Scipy-dev mailing list
http://projects.scipy.org/mailman/listinfo/scipy-dev
Michael McNeil Forbes
2007-04-13 00:55:48 UTC
Permalink
Post by Ondrej Certik
Could you Michael please also rewrite your code to Python?
http://www.phys.washington.edu/users/mforbes/projects/broyden/
It would be nice to have all of it in SciPy with the same interface.
BTW - why are you using matlab at all? To me, the python with numpy is
better than anything else I've programmed in, including matlab.
I wrote this code back when I had easy access to matlab and before I
knew
python. I simply have not had time to port the broyden code to python
yet.
Hopefully I will find time soon (it would also be nice to organize the
pieces in a modular fashion as Matthieu suggested, but I have simply not
had time to look over that yet.)

There are still a few things I miss about matlab, especially a good
line-by-line profiler, but generally I agree that python+numpy is
much better for programming.

Michael.
Bill Baxter
2007-04-13 02:32:41 UTC
Permalink
Post by Ondrej Certik
Could you Michael please also rewrite your code to Python?
http://www.phys.washington.edu/users/mforbes/projects/broyden/
It would be nice to have all of it in SciPy with the same interface.
BTW - why are you using matlab at all? To me, the python with numpy is
better than anything else I've programmed in, including matlab.
I don't know if you were serious or not, but Matlab still has a huge
amount of inertia and a very good network effect going for it. Plus
many more years of development behind it than numpy. You're much more
likely to be able to find random algorithms on the net implemented in
Matlab than in Python/Numpy, that is if you even need to go looking,
because a lot of stuff is already in Matlab to begin with.

So while I personally agree that Numpy is better than Matlab codewise,
a code in the hand beats two in the bush. I'd be very happy to drop
Matlab completely if you could just port the NetLab
(http://www.ncrg.aston.ac.uk/netlab/) and a few other things for me.
:-)

--bb
David Cournapeau
2007-04-13 02:53:56 UTC
Permalink
Post by Bill Baxter
I don't know if you were serious or not, but Matlab still has a huge
amount of inertia and a very good network effect going for it. Plus
many more years of development behind it than numpy. You're much more
likely to be able to find random algorithms on the net implemented in
Matlab than in Python/Numpy, that is if you even need to go looking,
because a lot of stuff is already in Matlab to begin with.
So while I personally agree that Numpy is better than Matlab codewise,
a code in the hand beats two in the bush. I'd be very happy to drop
Matlab completely if you could just port the NetLab
(http://www.ncrg.aston.ac.uk/netlab/) and a few other things for me.
Well, netlab is a huge package, but google just announced that my SoC
projet pymachine, a set of toolboxes for machine learning-related
algorithms, has been accepted:

http://code.google.com/soc/psf/appinfo.html?csaid=44CD86A83707638A (the
full proposal can be found there:
http://www.ar.media.kyoto-u.ac.jp/members/david/fullproposal.html)

So expect some news (and more importantly, some code) on this front in
the coming months,

cheers,

David
Bill Baxter
2007-04-13 03:13:34 UTC
Permalink
Post by David Cournapeau
Well, netlab is a huge package, but google just announced that my SoC
projet pymachine, a set of toolboxes for machine learning-related
That's great to hear. I've been keeping an eye on your progress with
pyEM and such. Very promising.

Incidentally I work with some speech and graphics guys from ATR, where
I see you worked previously. Do you know Yotsukura-san, Kawamoto-san,
or Nakamura-san? (I think Nakamura-san is now the head of ATR maybe
even).
Post by David Cournapeau
http://code.google.com/soc/psf/appinfo.html?csaid=44CD86A83707638A (the
http://www.ar.media.kyoto-u.ac.jp/members/david/fullproposal.html)
So expect some news (and more importantly, some code) on this front in
the coming months,
I would be interested in joining a dev list on this or something like
that (or open dev blog? or wiki?) if you start such a thing. I assume
you have to have discussions with your mentor anyway. If possible
it'd be nice to peek in on those conversations.

--bb
Robert Kern
2007-04-13 03:20:08 UTC
Permalink
Post by Bill Baxter
Post by David Cournapeau
http://code.google.com/soc/psf/appinfo.html?csaid=44CD86A83707638A (the
http://www.ar.media.kyoto-u.ac.jp/members/david/fullproposal.html)
So expect some news (and more importantly, some code) on this front in
the coming months,
I would be interested in joining a dev list on this or something like
that (or open dev blog? or wiki?) if you start such a thing. I assume
you have to have discussions with your mentor anyway. If possible
it'd be nice to peek in on those conversations.
David is welcome to use scipy-dev and scipy.org especially since a good chunk of
the project involves scipy packages.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
David Cournapeau
2007-04-13 04:59:49 UTC
Permalink
Post by Bill Baxter
Post by David Cournapeau
Well, netlab is a huge package, but google just announced that my SoC
projet pymachine, a set of toolboxes for machine learning-related
That's great to hear. I've been keeping an eye on your progress with
pyEM and such. Very promising.
Incidentally I work with some speech and graphics guys from ATR, where
I see you worked previously. Do you know Yotsukura-san, Kawamoto-san,
or Nakamura-san? (I think Nakamura-san is now the head of ATR maybe
even).
I indeed work some time there before starting my PhD program at Kyodai,
but not in the speech lab (I worked in the now defunct MIS lab).
Post by Bill Baxter
I would be interested in joining a dev list on this or something like
that (or open dev blog? or wiki?) if you start such a thing. I assume
you have to have discussions with your mentor anyway. If possible
it'd be nice to peek in on those conversations.
There is nothing started yet, and some things need to be fixed with my
mentor before things get started, but as Robert said, most if not all
discussion related to it would happen here and follow the usual scipy
process (scipy SVN, Trac, etc...).

David
Bill Baxter
2007-04-13 05:41:10 UTC
Permalink
Post by David Cournapeau
Post by Bill Baxter
I would be interested in joining a dev list on this or something like
that (or open dev blog? or wiki?) if you start such a thing. I assume
you have to have discussions with your mentor anyway. If possible
it'd be nice to peek in on those conversations.
There is nothing started yet, and some things need to be fixed with my
mentor before things get started, but as Robert said, most if not all
discussion related to it would happen here and follow the usual scipy
process (scipy SVN, Trac, etc...).
Great then.

The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?

The basic components of this stuff (like RBFs) also make for good
scattered data interpolation schemes. I hear questions every so often
on the list about good ways to do that, so making the tools for the
machine learning toolkit easy to use for people who just want to
interpolate data would be nice.

Going in a slightly different direction, meshfree methods for solving
partial differential equations also build on tools like RBF and moving
least squares interpolation. So for that reason too, it would be nice
to have a reusable api layer for those things.

You mention also that you're planning to unify row vec vs. column vec
conventions. Just wanted to put my vote in for row vectors! For a
number of reasons
1) It seems to be the more common usage in machine learning literature
2) with Numpy's default C-contiguous data it puts individual vectors
in contiguous memory.
3) it's easier to print something that's Nx5 than 5xN
4) "for vec in lotsofvecs:" works with row vectors, but needs a
transpose for column vectors.
5) accessing a vector becomes just data[i] instead of data[:,i] which
makes it easier to go back and forth between a python list of vectors
and a numpy 2d array of vectors.

--bb
Bill Baxter
2007-04-13 05:44:37 UTC
Permalink
Post by Bill Baxter
You mention also that you're planning to unify row vec vs. column vec
conventions. Just wanted to put my vote in for row vectors! For a
number of reasons
1) It seems to be the more common usage in machine learning literature
2) with Numpy's default C-contiguous data it puts individual vectors
in contiguous memory.
3) it's easier to print something that's Nx5 than 5xN
4) "for vec in lotsofvecs:" works with row vectors, but needs a
transpose for column vectors.
5) accessing a vector becomes just data[i] instead of data[:,i] which
makes it easier to go back and forth between a python list of vectors
and a numpy 2d array of vectors.
One more:
6) mat(avec) where avec is 1-D returns a row vector rather than a column vector.
Robert Kern
2007-04-13 05:50:54 UTC
Permalink
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
On that note, I have some Gaussian process code in a Mercurial repository here
(click the "manifest" button to browse the source):

http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/gp/

It's based on the treatment by Rasmussen and Williams:

http://www.gaussianprocess.org/gpml/

The covariance functions I implement there might be useful in other methods, too.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
David Cournapeau
2007-04-13 06:08:14 UTC
Permalink
Post by Robert Kern
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
On that note, I have some Gaussian process code in a Mercurial repository here
http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/gp/
http://www.gaussianprocess.org/gpml/
The covariance functions I implement there might be useful in other methods, too.
Thanks for those links, I will take a look at the code you wrote.

Since you're here, I have some questions concerning chaco for the
visualization part of the project. Basically, I am unsure about whether
I should use chaco or matplotlitb. I do not know chaco very well yet,
but it seems much better API and performance wise compared to matplotlib
for interactive visualization. The problem is that it still does not
have a lot of visibility to the community compared to matplotlib, and it
is still pretty complicated to install. I do not care much about those
points myself, but seeing how installation problems are one of the big
difficulty for newcommers to numpy/scipy, I am a bit concerned. Is this
impression funded, and if it is, is there a chance to see improvements
on those fronts in the next few months ?

David
Robert Kern
2007-04-13 07:49:17 UTC
Permalink
Post by David Cournapeau
Since you're here, I have some questions concerning chaco for the
visualization part of the project. Basically, I am unsure about whether
I should use chaco or matplotlitb. I do not know chaco very well yet,
but it seems much better API and performance wise compared to matplotlib
for interactive visualization. The problem is that it still does not
have a lot of visibility to the community compared to matplotlib, and it
is still pretty complicated to install. I do not care much about those
points myself, but seeing how installation problems are one of the big
difficulty for newcommers to numpy/scipy, I am a bit concerned. Is this
impression funded, and if it is, is there a chance to see improvements
on those fronts in the next few months ?
Just installing Chaco and the stuff it depends on can actually be somewhat
easier than matplotlib: we don't bother with external libjpeg, libpng, and
libfreetype libraries. The major issue would actually be disabling building of
TVTK if you don't have VTK installed and you only care about Chaco, but even
that's a comment-out-one-line operation.

In the coming weeks, though, we will be playing around with a reorganization of
the repository along the lines of the scikits layout if you've been following
that conversation. That would enable one to just build the enthought subpackages
that you need, or allow easy_install to do so. Even if we don't end up
reorganizing the trunk that way, we'll probably have such a reorganized mirror
of the trunk using svn:externals to get the same effect for distribution purposes.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
David Cournapeau
2007-04-13 05:57:31 UTC
Permalink
Post by Bill Baxter
Post by David Cournapeau
Post by Bill Baxter
I would be interested in joining a dev list on this or something like
that (or open dev blog? or wiki?) if you start such a thing. I assume
you have to have discussions with your mentor anyway. If possible
it'd be nice to peek in on those conversations.
There is nothing started yet, and some things need to be fixed with my
mentor before things get started, but as Robert said, most if not all
discussion related to it would happen here and follow the usual scipy
process (scipy SVN, Trac, etc...).
Great then.
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
The plan is really about unifying and improving existing toolboxes, and
provides a higher level API (which would end up in scikits for various
reasons). Depending on the time left, I will add some algorithms later.
Of course, the goal is that other people will also jump in to add new
algorithms (I for example will add some recent advances for mixture like
ensemble learning, outside of the SoC if necessary).
Post by Bill Baxter
The basic components of this stuff (like RBFs) also make for good
scattered data interpolation schemes. I hear questions every so often
on the list about good ways to do that, so making the tools for the
machine learning toolkit easy to use for people who just want to
interpolate data would be nice.
Going in a slightly different direction, meshfree methods for solving
partial differential equations also build on tools like RBF and moving
least squares interpolation. So for that reason too, it would be nice
to have a reusable api layer for those things.
You mention also that you're planning to unify row vec vs. column vec
conventions. Just wanted to put my vote in for row vectors! For a
number of reasons
1) It seems to be the more common usage in machine learning literature
2) with Numpy's default C-contiguous data it puts individual vectors
in contiguous memory.
3) it's easier to print something that's Nx5 than 5xN
4) "for vec in lotsofvecs:" works with row vectors, but needs a
transpose for column vectors.
5) accessing a vector becomes just data[i] instead of data[:,i] which
makes it easier to go back and forth between a python list of vectors
and a numpy 2d array of vectors.
I have not given a lot of thoughts about it yet; what matters the most
is that all algo have the same conventions. Nevertheless, my experience
so far in numpy is similar to yours with regard to ML algorithms (except
point 2: depending on the algo. you need contiguous access along one
dimension, and my impression is that in numpy, this matters a lot
performance wise, at least much more than in matlab).

David
Travis Oliphant
2007-04-13 07:06:41 UTC
Permalink
Post by David Cournapeau
I have not given a lot of thoughts about it yet; what matters the most
is that all algo have the same conventions. Nevertheless, my experience
so far in numpy is similar to yours with regard to ML algorithms (except
point 2: depending on the algo. you need contiguous access along one
dimension, and my impression is that in numpy, this matters a lot
performance wise, at least much more than in matlab).
If I understand you correctly, this is not as true as it once was.
There was some benchmark code that showed a slow-down on transposed
arrays until we caught the bug that was causing it.

What is important is that your inner-loops are running over data that is
"closest" together in memory. Exactly how close is close enough
depends on cache size.

-Travis
Matthieu Brucher
2007-04-13 06:23:12 UTC
Permalink
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
Don't scipy have SVMs already ? Perhaps not as modularized at it could be ?
PPCA is PCA IIRC (Tipping 97, it's part of my Phd thesis), KPCA is not a big
deal if kernels are in a module, and if they have a method taking 2
arguments. BTW, even the svd could directly take a kernel as an argument,
the default kernel being the scalar product ?
I'm in favour of fine-grained modules - like for the optimisation module I
proposed -, and allowing pepole to choose which kernel they want, even if
the kernel was designed for SVM, is a good thing, the "kernel trick" should
be almost universal :)

Matthieu
David Cournapeau
2007-04-13 06:40:18 UTC
Permalink
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
Don't scipy have SVMs already ? Perhaps not as modularized at it could be ?
PPCA is PCA IIRC (Tipping 97, it's part of my Phd thesis), KPCA is not
a big deal if kernels are in a module, and if they have a method
taking 2 arguments. BTW, even the svd could directly take a kernel as
an argument, the default kernel being the scalar product ?
I'm in favour of fine-grained modules - like for the optimisation
module I proposed -, and allowing pepole to choose which kernel they
want, even if the kernel was designed for SVM, is a good thing, the
"kernel trick" should be almost universal :)
The project is first about unifying *existing* packages: basically, make
them first class citizen doc-wise and api-wise, so that they can be
moved out of the sandbox. SVM, EM for mixture fall in this category.

cheers,

David
Bill Baxter
2007-04-13 06:46:27 UTC
Permalink
Post by Matthieu Brucher
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
Don't scipy have SVMs already ? Perhaps not as modularized at it could be ?
It might have something in sandbox, but as far as I'm concerned 'in
scipy.sandbox' is synonymous with 'not in scipy'.
Post by Matthieu Brucher
PPCA is PCA IIRC (Tipping 97, it's part of my Phd thesis),
Yes, pretty much so, just with some variances added into the diagonal
of the matrix at the right place.
Post by Matthieu Brucher
KPCA is not a big
deal if kernels are in a module, and if they have a method taking 2
arguments.
Right. Low level kernel stuff in reusable lib makes a lot of sense.
If there were a lib of functions with common kernels and their
derivatives and possibly even 2nd derivatives, that would cover a lot
of territory. (I could use some 2nd derivatives right now...)

--bb
Matthieu Brucher
2007-04-13 07:11:42 UTC
Permalink
Post by Bill Baxter
Post by Matthieu Brucher
Don't scipy have SVMs already ? Perhaps not as modularized at it could
be ?
It might have something in sandbox, but as far as I'm concerned 'in
scipy.sandbox' is synonymous with 'not in scipy'.
OK, I read David's answer - BTW, hello Gabou :) -. I'm looking forward to
this, I will probably use SVMs in a near future, as I'm moving toward
Python.
Post by Bill Baxter
PPCA is PCA IIRC (Tipping 97, it's part of my Phd thesis),
Yes, pretty much so, just with some variances added into the diagonal
of the matrix at the right place.
OK, what you want is in fact the implementation of the optimization
algorithm given at the end of the paper ? Because in the standard form of
PPCA, the error variance is isotropic, and in that case, it is tantamount to
simple PCA.
Post by Bill Baxter
KPCA is not a big
Post by Matthieu Brucher
deal if kernels are in a module, and if they have a method taking 2
arguments.
Right. Low level kernel stuff in reusable lib makes a lot of sense.
If there were a lib of functions with common kernels and their
derivatives and possibly even 2nd derivatives, that would cover a lot
of territory. (I could use some 2nd derivatives right now...)
I would appreciate as well, for a modified mean-shift algorithm.
I'll probably port Isomap and LLE algorithms as well, they are widely used
for manifold learning (they can be expressed as KPCA algorithms with a
particular kernel).


Matthieu
Matthieu Brucher
2007-04-17 08:12:12 UTC
Permalink
Post by Bill Baxter
The project page mentions SVM. In addition to SVM I'm interested in
things like PPCA, kernel PCA, RBF networks, gaussian processes and
GPLVM. Are you going to try to go in the direction of a modular
structure with reusable bits for for all kernel methods, or is the
plan to targeted specifically SVM?
There is a review of dimensionality reduction algorithm on
http://www.cs.unimaas.nl/l.vandermaaten/Laurens%20van%20der%20Maaten/Matlab%20Toolbox%20for%20Dimensionality%20Reduction.html
that could be woth porting... one day...

Matthieu
Joe Harrington
2007-04-12 18:22:18 UTC
Permalink
Just a comment on Robert's otherwise-excellent reply, we agreed some
time ago that the forum for discussing changes to the site is this
list, not trac. This is because many participants such as myself are
not involved in code development and do not have (or need) trac
accounts. We should *not* encourage people simply to romp in the
pages and restructure as they please, as the web site is in constant
public view by more than just developers and should therefore not be a
playground for testing ideas (except for DevZone, which is
specifically for that purpose). Shortly after the switch to the Moin
site, someone went in and rewrote a bunch of the pages to follow their
own style, and it made us realize that an open invitation to edit was
not the best idea.

Small changes like adding a link or an entry in a list are of course
fine to make. Changing a page's overall structure should at least get
a brief review by the list. The page layouts are simple enough that
improvements can either be discussed based on a posted description, or
actually made by example. For the latter, copy the page onto a page
hanging off of DevZone and post an email pointing to it and asking for
comment. For obvious reasons, the front page can only be modified by
a few people, not just anyone with an account. It would be best if
people making regular changes identified themselves in DevZone as site
maintainers so that others can find them.

That said, I agree restructuring is called for in some cases, and as
Robert pointed out, in the doc area what's really needed is a doc. I
think we'll be quick to cheer on anything reasonable in either area.

--jh--
Robert Kern
2007-04-12 21:06:55 UTC
Permalink
Post by Joe Harrington
Just a comment on Robert's otherwise-excellent reply, we agreed some
time ago that the forum for discussing changes to the site is this
list, not trac. This is because many participants such as myself are
not involved in code development and do not have (or need) trac
accounts.
I don't actually recall any agreement to that effect. The Trac exists for
tracking issues for all parts of the project, not just bugs in the code. If you
want to be active in the project even you are not a developer of code, use the Trac.
Post by Joe Harrington
We should *not* encourage people simply to romp in the
pages and restructure as they please, as the web site is in constant
public view by more than just developers and should therefore not be a
playground for testing ideas (except for DevZone, which is
specifically for that purpose). Shortly after the switch to the Moin
site, someone went in and rewrote a bunch of the pages to follow their
own style, and it made us realize that an open invitation to edit was
not the best idea.
Again, I'm pretty sure that there was no such universal realization.

I explicitly do encourage people to use the Wiki and change pages as they see
fit (except for FrontPage because it's special). If you are concerned about
changes, watch the RSS feed and change things back if you disagree with the
changes. If there is continued disagreement, then bring it to the list. That's
how Wikis are supposed to work.

We have no lack of suggestions on this list about how a page ought to look or
what it ought to have. What we lack are people actually willing to put the time
in to do the edits and provide the content. I encourage the latter; the former
needs no such encouragement.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Continue reading on narkive:
Loading...