[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

Hi Karan,

Glad to see interest in this project!

Your credentials look great. We'll need a small patch to numpy or scipy,
too.

A few scattered thoughts about the project:

- This is not exactly a transition to pytest as such. The goal is to allow
an alternative test runner, with minimal changes to the test suite.
- At the moment, nose is an optional dependency of numpy and scipy. This
should be the same with pytest: both numpy and scipy should remain
importable without either nose or pytest.
- All tests should be run with either test runner (this will need to be
checked early on).
- We should try to decouple the test runner as much as reasonable.
pytest-specific things should be kept to a minimum. E.g. rewriting of
assertions should be left out, I think.

A few possibly sticky points:

* both test suites are organized per submodule. This does not seem to mesh
well with pytest's idea of collecting things in a conftest.py file: we
either need to duplicate conftest.py files in each submodule/tests/ folder
or there hopefully is some better magic. At least I did not manage to in a
quick look I took a while ago, https://github.com/ev-br/numpy/commits/pytest

* yield-based test generators. pytest says they are deprecated and are
going to be removed in pytest 4. I don't know what is the status of pytest
4 --- is it something with a defined release date or just some uncertain
future horizon or something in between. I guess we'll need to ask pytest
devs. Meanwhile, there is a switch to silence the deprecation warnings in
pytest 3.

* The solution should work for both numpy and scipy. It seems reasonable to
hide the nose/pytest specific stuff in the numpy.testing module. However
scipy (and other downstream packages) have to support older versions of
numpy, so anything new in numpy 1.13 is not going to be immediately useful
downstream.

It would also be good to hear from numpy devs (which I am not, so take what
I'm saying about numpy with a spoonful of salt :-))

Cheers,

Evgeni

Post by Karan Desai
Hello developers,
I am Karan Desai, a third year undergraduate from Indian Institute of
technology, Roorkee. I went through the GSoC Ideas page on Scipy github
wiki, and I think that I'll be most comfortable with the idea "Make the
SciPy test suite work with pytest".
I believe that I match the required prerequisites, and I'll present some
facts to support my belief.
1. I have completed my GSoC'16 with TARDIS. Their integration tests
took more than a day to execute, hence Travis was not fit for that. I
developed an Integration Testing Framework in *pytest* to schedule
their execution on a local server when a commit occurs on Github. Also, it
generated an automated HTML Report with comparison plots between various
astronomical quantities in order to debug failed tests.
2. I have contributed to *joblib*, a multiprocessing library in
Python. I have helped with the migration of joblib testing framework from
nose to pytest. Including review cycles, it took about 25 days to do so.
Scipy and numpy will be bigger than that.
With sufficient knowledge of how pytest flavored tests are designed, and
some important differences between nose and pytest styles, I think I'll be
very much comfortable with this project.
I saw recent mail threads in the archive and I was happily surprised to
see a few familiar names, Gael Varoquaux and Olivier Grisel might have
remembered my contributions in Joblib back in DecemberâŠ
1. Github Profile: https://www.github.com/karandesai-96/
https://www.github.com/joblib/joblib/issues/411
<https://www.github.com/joblib/joblib/issues/411>
3. GSoC 2016 project: https://summerofcode.withgoogle.com/archive/2016/
projects/4796129849901056/
I will be happy to hear from the community.
Thanks.
Karan Desai,
Department of Electrical Engineering,
IIT Roorkee.
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev

Charles R Harris

2017-02-27 16:44:29 UTC

Post by Evgeni Burovski
Hi Karan,
Glad to see interest in this project!
Your credentials look great. We'll need a small patch to numpy or scipy,
too.
- This is not exactly a transition to pytest as such. The goal is to allow
an alternative test runner, with minimal changes to the test suite.
- At the moment, nose is an optional dependency of numpy and scipy. This
should be the same with pytest: both numpy and scipy should remain
importable without either nose or pytest.
- All tests should be run with either test runner (this will need to be
checked early on).
- We should try to decouple the test runner as much as reasonable.
pytest-specific things should be kept to a minimum. E.g. rewriting of
assertions should be left out, I think.
* both test suites are organized per submodule. This does not seem to mesh
well with pytest's idea of collecting things in a conftest.py file: we
either need to duplicate conftest.py files in each submodule/tests/ folder
or there hopefully is some better magic. At least I did not manage to in a
quick look I took a while ago, https://github.com/ev-br/
numpy/commits/pytest
* yield-based test generators. pytest says they are deprecated and are
going to be removed in pytest 4. I don't know what is the status of pytest
4 --- is it something with a defined release date or just some uncertain
future horizon or something in between. I guess we'll need to ask pytest
devs. Meanwhile, there is a switch to silence the deprecation warnings in
pytest 3.
* The solution should work for both numpy and scipy. It seems reasonable
to hide the nose/pytest specific stuff in the numpy.testing module. However
scipy (and other downstream packages) have to support older versions of
numpy, so anything new in numpy 1.13 is not going to be immediately useful
downstream.
It would also be good to hear from numpy devs (which I am not, so take
what I'm saying about numpy with a spoonful of salt :-))

I think we need to redo the generator based tests to not use generators. If
we confine numpy/scipy testing to pytest we may look at parametrized tests
as a replacement. Supporting both pytest and nose for downstream projects
may be tricky. Note that the downstream projects already using pytest don't
need us, so the number requiring the numpy testing apparatus may be limited.

I don't know much about pytest, so suggestions from more knowledgeable
folks is welcome.

<snip>

Chuck

Pauli Virtanen

2017-02-27 21:03:27 UTC

Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.

I think retaining nose compatibility in the numpy/scipy test suites does
not bring advantages --- if we are planning to retain it, I would like to
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.

I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think not
many remember how it works in depth.

I think the needs for Scipy and Numpy are not demanding, so sticking with
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but AFAIK
the ones in numpy.testing are pytest compatible (and independent of the
nose machinery).

For numpy.testing, retaining nose backward compatibility is important.
However, for the scipy/numpy test suites this seems less important to me.

Thoughts?

Best,
Pauli

Evgeni Burovski

2017-02-27 21:35:04 UTC

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.
I think retaining nose compatibility in the numpy/scipy test suites does
not bring advantages --- if we are planning to retain it, I would like to
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.
I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think not
many remember how it works in depth.
I think the needs for Scipy and Numpy are not demanding, so sticking with
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but AFAIK
the ones in numpy.testing are pytest compatible (and independent of the
nose machinery).
For numpy.testing, retaining nose backward compatibility is important.
However, for the scipy/numpy test suites this seems less important to me.
Thoughts?

My reasoning was simple: there is already a compatibility layer of
numpy.testing.

As soon as numpy.testing works in a virtualenv which has pytest and
does not have nose, the scipy test suite just works, modulo maybe a
handful of nose-isms to clean up (a stray import from nose.tools, a
setUp/tearDown pair in a class derived from object etc). To catch
those, we anyway need to have two parallel installations, one with
nose, and one without, at least temporarily.

At this point we can either stop or do additional work of removing
nose from numpy.testing, which is invisible for scipy --- because
numpy.testing just shields it all.

I certainly don't understand all details of numpy.testing. That said,
ISTM that we only need to port a few bits: submodule.test() and
np.testing.run_module_suite.

This of course leaves two elephants in the room:

- backwards compat for older numpy versions. A straightforward
solution is to make a separate package, numpy-testing and mechanically
import from it instead of numpy.testing (or make the latter import
from the former).
- yield based test generators.

Evgeni

Pauli Virtanen

2017-02-27 22:30:18 UTC

Tue, 28 Feb 2017 00:35:04 +0300, Evgeni Burovski kirjoitti:
[clip]

Post by Evgeni Burovski

Post by Pauli Virtanen
Thoughts?

I'm actually advocating for abandoning numpy.testing (ie. the test runner
part --- the assert functions are fine). I think this is what other numpy-
dependent projects that use pytest do. Making numpy.testing test runners
work with pytest sounds like a complication that's not really necessary,
and adding up more things to maintain.

If we are going to port the test suite to work with pytest, I would go
all the way. By "vanilla" pytest, I mean whatever pytest supports out of
the box, without inventing a new framework with custom conventions.

Post by Evgeni Burovski
- backwards compat for older numpy versions. A straightforward solution
is to make a separate package, numpy-testing and mechanically import
from it instead of numpy.testing (or make the latter import from the
former).
- yield based test generators.

I think the first issue goes away if we abandon numpy.testing --- the
nose-dependent parts will retained unchanged and maybe deprecated in the
long run, and the nose-independent parts kept.

The second issue probably doesn't --- these tests would have to be
rewritten with the corresponding pytest-isms (fixtures afaics) in any
case.

Pauli

j***@gmail.com

2017-02-27 22:49:44 UTC

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski

Post by Pauli Virtanen
Thoughts?

Is there still a pytest equivalent to running the test from inside an
interpreter?
import scipy.stats
scipy.stats.test()

I always liked this as a user (when I'm not already in a testing/debugging
shell).

Josef

Post by Pauli Virtanen
If we are going to port the test suite to work with pytest, I would go
all the way. By "vanilla" pytest, I mean whatever pytest supports out of
the box, without inventing a new framework with custom conventions.

I think the first issue goes away if we abandon numpy.testing --- the
nose-dependent parts will retained unchanged and maybe deprecated in the
long run, and the nose-independent parts kept.
The second issue probably doesn't --- these tests would have to be
rewritten with the corresponding pytest-isms (fixtures afaics) in any
case.
Pauli
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev

Robert Kern

2017-02-27 21:35:54 UTC

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.
I think retaining nose compatibility in the numpy/scipy test suites does
not bring advantages --- if we are planning to retain it, I would like to
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.
I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think not
many remember how it works in depth.
I think the needs for Scipy and Numpy are not demanding, so sticking with
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but AFAIK
the ones in numpy.testing are pytest compatible (and independent of the
nose machinery).

If we're migrating test runners, I'd rather drop all dependencies and
target `python -m unittest discover` as the lowest common denominator
rather than target a fuzzy "vanilla" subset of pytest in particular.

This may well be the technical effect of what you are describing, but I
think it's worth explicitly stating that focus.

--
Robert Kern

Charles R Harris

2017-02-27 22:32:36 UTC

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.
I think retaining nose compatibility in the numpy/scipy test suites does
not bring advantages --- if we are planning to retain it, I would like to
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.
I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think not
many remember how it works in depth.
I think the needs for Scipy and Numpy are not demanding, so sticking with
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but AFAIK
the ones in numpy.testing are pytest compatible (and independent of the
nose machinery).

I'd certainly expect to make full use of the pytest features, why use it
otherwise? There is a reason that both nose and pytest extended unittest.
The only advantage I see to unittest is that it is less likely to become
abandon ware.

Post by Robert Kern
This may well be the technical effect of what you are describing, but I
think it's worth explicitly stating that focus.

I don't think that us where we were headed. Can you make the case for
unittest?

Chuck

Robert Kern

2017-02-27 23:08:34 UTC

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.
I think retaining nose compatibility in the numpy/scipy test suites does
not bring advantages --- if we are planning to retain it, I would like to
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.
I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think not
many remember how it works in depth.
I think the needs for Scipy and Numpy are not demanding, so sticking with
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but AFAIK
the ones in numpy.testing are pytest compatible (and independent of the
nose machinery).

If we're migrating test runners, I'd rather drop all dependencies and

target `python -m unittest discover` as the lowest common denominator
rather than target a fuzzy "vanilla" subset of pytest in particular.

Post by Charles R Harris
I'd certainly expect to make full use of the pytest features, why use it

otherwise? There is a reason that both nose and pytest extended unittest.
The only advantage I see to unittest is that it is less likely to become
abandon ware.

Well, that's definitely not what Pauli was advocating, and my response was
intending to clarify his position.

Post by Robert Kern
This may well be the technical effect of what you are describing, but I

think it's worth explicitly stating that focus.

Post by Charles R Harris
I don't think that us where we were headed. Can you make the case for

unittest?

There are two places where nose and pytest provide features; they are each
a *test runner* and also a *test framework*. As a test runner, they provide
features for discovering tests (essentially, they let you easily point to a
subset of tests to run) and reporting the test results in a variety of
user-friendly ways. Test discovery functionality works with
plain-old-TestCases, and pretty much any test runner works with the
plain-old-TestCases. These features face the user who is *running* the
tests.

The other place they provide features is as test frameworks. This is where
things like the generator tests and additional fixtures come in (i.e.
setup_class, setup_module, etc.). These require you to write your tests in
framework-specific ways. Those written for nose don't necessarily work with
those written for pytest and vice versa. And any other plain test runner is
right out. These features face the developer who is *writing* the tests.

Test discovery was the main reason we started using nose over plain
unittest. At that time (and incidentally, when nose and pytest began),
`python -m unittest discover` didn't exist. To run unit tests, you had to
manually collect the TestCases into a TestSuite and write your own logic
for configuring any subsets from user input. It was a pain in the ass for
large hierarchical packages like numpy and scipy. The test framework
features of nose like generator tests were nice bonuses, but were not the
main motivating factor. [Source: me; it was Jarrod and I who made the
decision to use nose at a numpy sprint in Berkeley.]

If we're going to make a switch, I'd rather give up those framework
features (we don't use all that many) in order to get agnosticism on the
test runner front. Being agnostic to the test runner lets us optimize for
multiple use cases simultaneously. That is, the best test runner for "run
the whole suite and record every detail with coverage under time
constraints under Travis CI" is often different from what a developer wants
for "run just the one test module for the thing I'm working on and report
to me only what breaks as fast as possible with minimal clutter".

--
Robert Kern

Ralf Gommers

2017-02-28 09:31:26 UTC

On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris <

Post by Pauli Virtanen
[clip]

Post by Evgeni Burovski
- This is not exactly a transition to pytest as such. The goal is to
allow an alternative test runner, with minimal changes to the test
suite.

I'd suggest we should consider actually dropping nose, if we make the
suite pytest compatible.
I think retaining nose compatibility in the numpy/scipy test suites

does

Post by Pauli Virtanen
not bring advantages --- if we are planning to retain it, I would

like to

Post by Pauli Virtanen
understand why. Supporting two systems is more complicated, makes it
harder to add new tests, and if the other one is not regularly used,
it'll probably break often.
I think we should avoid temptation to build compatibility layers, or
custom test runners --- there's already complication in numpy.testing
that IIRC originates from working around issues in nose, and I think

not

Post by Pauli Virtanen
many remember how it works in depth.
I think the needs for Scipy and Numpy are not demanding, so sticking

with

Post by Pauli Virtanen
"vanilla" pytest features and using only its native test runner would
sound best. The main requirement is numerical assert functions, but

AFAIK

Post by Pauli Virtanen
the ones in numpy.testing are pytest compatible (and independent of

the

Post by Pauli Virtanen
nose machinery).

If we're migrating test runners, I'd rather drop all dependencies and

target `python -m unittest discover` as the lowest common denominator
rather than target a fuzzy "vanilla" subset of pytest in particular.

Post by Charles R Harris
I'd certainly expect to make full use of the pytest features, why use it

otherwise? There is a reason that both nose and pytest extended unittest.
The only advantage I see to unittest is that it is less likely to become
abandon ware.
Well, that's definitely not what Pauli was advocating, and my response was
intending to clarify his position.