[SciPy-Dev] [GSoC 2017] Nose to Pytest Migration - conclusions from previous thread

[Karan Desai]

Hello developers,
I wrote an introductory thread about two weeks ago, expressing my interest in
this project. The thread grew very huge and it took me a while to aggregate the
suggestions and views of all the fellow developers of the community. Meanwhile,
I also inspected the test suite of scipy. I am putting up my conclusions and
thoughts further.
As an added information, I should specify, as earlier that I have dropped nose
and integrated pytest properly into "joblib". Some of the test suite designs
were positively affirmed by a Pytest core developer on the issue thread. Here
they go:
1. The homepage of Nose after its release of 1.3.7 states that it is no longer
maintained. I see no reason continue supporting it, as now or later we will
have to let it go. Also I am sure many prominent libraries having scipy or
numpy as dependency have already started the shift.


2. If we are using pytest, then we should make full usage of pytest alone. A
small utility library like joblib with a few hunderds of tests cut down
around 700 lines of code by adopting pytest design.


3. While testing utils can be kept independent of pytest, we should fully
embrace pytest for own own unit tests. or even better, we can thinly wrap
pytest and make array assertion possible by plain assert keyword statements
in some way. ( I haven't thought about it yet ).


4. Making the test suites just work with pytest is a matter of one PR, or a
weekend's work and I agree with developers of scikit-image. But it takes
more time to complete redesign tests in flavor of pytest, after all we have
to think about future contributors joining in. People experienced with
pytest should fine the test suite design familiar, for contributions. I
think that scikit-image has achieved the former, but still might need some
PRs for the latter ( I'm glad to help in future ).


5. Here are some notable changes which are required for pytest transition:
6. Plain assert keyword statements.Pytest's own raise and warn assert helpers
produce cleaner error logs on failure.Use pytest's parametrization to get
rid of yield based tests and form compact tests.Fixtures like tmpdir,
capsys, monkeypatch and others provided by pytest help reduce a lot of
boiler plate code.


7. Analyzing the test suite of scipy, I think that it is perfectly fine to
adopt a strategy I used before in Joblib. I will present my first draft
soon, and it will be heavily inspired from this issue thread:
https://www.github.com/joblib/joblib/issues/411 . I made around 20 PRs and
after merge of every PR, the suite was in a working condition, so keeping a
separate branch shall not be required.

I suggest on not falling back to unittests as it will increase the boilerplate
code heavily. The reason of not supporting both nosetests and pytest is because
they have different flavours, and that it would be difficult to keep up with
nose an pytest always comes with something interesting every two months in new
releases.
I will be happy to hear and work upon further suggestions. Thanks.
Regards,Karan.

[Robert Kern]

Post by Karan Desai
While testing utils can be kept independent of pytest, we should fully

embrace pytest for own own unit tests. or even better, we can thinly wrap
pytest and make array assertion possible by plain assert keyword statements
in some way. ( I haven't thought about it yet ).

I don't think it's possible, no. pytest and nose don't change the operation
of the `assert` statement at all; they just introspect the code and the
namespace when the assertion fails to provide a helpful message. `assert x
== y` just doesn't work when the arguments are arrays because the result is
not coerceable to a proper `bool`.

Post by Karan Desai
Making the test suites just work with pytest is a matter of one PR, or a

weekend's work and I agree with developers of scikit-image. But it takes
more time to complete redesign tests in flavor of pytest, after all we have
to think about future contributors joining in. People experienced with
pytest should fine the test suite design familiar, for contributions. I
think that scikit-image has achieved the former, but still might need some
PRs for the latter ( I'm glad to help in future ).

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`
statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I
forget if we've made that a policy there too.

So if you mean, by this requirement, that we convert all of our
`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.

--
Robert Kern

[Nathaniel Smith]

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`
statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I
forget if we've made that a policy there too.

So if you mean, by this requirement, that we convert all of our
`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.


Pytest arranges for assert statements in test modules to be run even if
Python has assert statements disabled in general. See

http://doc.pytest.org/en/latest/announce/release-2.1.0.html

-n

[j***@gmail.com]

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert` statements.
numpy's test suite must be runnable under `python -O`. numpy does some
docstring manipulation that could fail in principle under those conditions
(i.e. it did fail at one point until we fixed it, so now we have to ensure
that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I forget
if we've made that a policy there too.
So if you mean, by this requirement, that we convert all of our
`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.
Pytest arranges for assert statements in test modules to be run even if
Python has assert statements disabled in general. See
http://doc.pytest.org/en/latest/announce/release-2.1.0.html

sounds way too magic to me

I like the numpy asserts,
no statements just nice functions with options instead of magic code rewriting.

(and assert_allclose has better defaults than np.allclose)

Josef

-n
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev

[Robert Kern]

Post by Robert Kern

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`

statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I
forget if we've made that a policy there too.

Post by Robert Kern
So if you mean, by this requirement, that we convert all of our

`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.

Post by Robert Kern
Pytest arranges for assert statements in test modules to be run even if

Python has assert statements disabled in general. See

Post by Robert Kern
http://doc.pytest.org/en/latest/announce/release-2.1.0.html

Neat!

Nonetheless, converting the current tests to use `assert` statements is
hardly a requirement, just something that might be enabled by the
transition.

--
Robert Kern

[Evgeni Burovski]

Post by Robert Kern

Post by Robert Kern

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`
statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I forget
if we've made that a policy there too.
So if you mean, by this requirement, that we convert all of our
`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.
Pytest arranges for assert statements in test modules to be run even if
Python has assert statements disabled in general. See
http://doc.pytest.org/en/latest/announce/release-2.1.0.html

Neat!
Nonetheless, converting the current tests to use `assert` statements is
hardly a requirement, just something that might be enabled by the
transition.
--
Robert Kern

And since the subject is inevitably going to show up more then once, I
think the GSoC proposal should explicitly mention that assert
rewriting is not a part of the GSoC. And speaking of the proposal, I
think it's time to see the first draft.

Evgeni

[Karan Desai]

Okay, so I'll clarify one thing based on all the suggestions above:
Rewrting assert methods as statements will not be a part of my proposal. I am
talking about mainly two things here - one is about numpy testing utils, which
will be used by other libraries directly (like assert array methods), and other
is about the test suite of scipy and numpy itself, comprising of unit tests of
these two libraries itself.
My idea is to keep the former as much independent of testing framework as
possible, while adopt pytest flavor (viz. fixtures and parametrization) for the
latter.

 And speaking of the proposal, I think it's time to see the first draft.

Sure, coming up soon.
Regards,Karan.

Post by Robert Kern

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`
statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I forget
if we've made that a policy there too.
So if you mean, by this requirement, that we convert all of our
`assert_equal(x, y)` calls to `assert x == y`, no, we won't be doing that,
even in the cases where it would be possible.
Pytest arranges for assert statements in test modules to be run even if
Python has assert statements disabled in general. See
http://doc.pytest.org/en/latest/announce/release-2.1.0.html

Neat!
Nonetheless, converting the current tests to use `assert` statements is
hardly a requirement, just something that might be enabled by the
transition.
--
Robert Kern

And since the subject is inevitably going to show up more then once, I

think the GSoC proposal should explicitly mention that assert

rewriting is not a part of the GSoC. And speaking of the proposal, I

think it's time to see the first draft.




Evgeni

_______________________________________________

SciPy-Dev mailing list

SciPy-***@scipy.org

https://mail.scipy.org/mailman/listinfo/scipy-dev

[Ralf Gommers]

Post by Karan Desai
Rewrting assert methods as statements will not be a part of my proposal. I
am talking about mainly two things here - one is about numpy testing utils,
which will be used by other libraries directly (like assert array methods),
and other is about the test suite of scipy and numpy itself, comprising of
unit tests of these two libraries itself.
My idea is to keep the former as much independent of testing framework as
possible, while adopt pytest flavor (viz. fixtures and parametrization) for
the latter.

That sounds right to me.

Ralf

[Ralf Gommers]

Post by Robert Kern

Post by Karan Desai
Plain assert keyword statements.

Just a note: numpy's test suite, at least, must not use `assert`
statements. numpy's test suite must be runnable under `python -O`. numpy
does some docstring manipulation that could fail in principle under those
conditions (i.e. it did fail at one point until we fixed it, so now we have
to ensure that it doesn't regress). scipy might have the same issue (e.g.
`scipy.special` ufuncs and `scipy.stats` distribution objects), but I
forget if we've made that a policy there too.

Yes, that's the policy - no plain assert statements anywhere.

Ralf