Test classes created and initialized regardless of parallelism level

[roded]

Hi,
I'm trying to get my Spock test suite to run in parallel with the following settings in my SpockConfig.groovy:

defaultSpecificationExecutionMode ExecutionMode.CONCURRENT
defaultExecutionMode ExecutionMode.SAME_THREAD
fixed(5)

I've noticed that even with a fixed parallelism, all the test classes are created and their setupSpecs are called when the test suite starts and that their execution is limited by the parallelism (or so that's what I have observed).

Is this the intended behavior of Spock?

My classes' setupSpecs create a WebDriver instance and I'd like to be able to postpone its creation until execution of the test spec starts.

Thanks!

[Vampire]

I cannot reproduce what you describe, can you knit an MCVE?

[Vampire]

Adding a addSetupSpecInterceptor instead causes more test specs to run in parallel than I desire (one or two). I suppose this is due to the limit being just on the setupSpec execution and not the feature execution as well.

That's why I did not say a setup spec interceptor, but a spec interceptor ;-)

[roded]

OK!
So you mean this:

    void visitSpec(SpecInfo spec) {
        spec.addInterceptor {
            threadPool.submit(it::proceed).get()
        }
    }

It seems like it's working... but then when calling get(), it crashes due to:

java.lang.NullPointerException: Cannot invoke method isAnnotationPresent() on null object
java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke method isAnnotationPresent() on null object
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at com.roded.FixJunit3108Extension.visitSpec_closure1(FixJunit3108Extension.groovy:27)
	at groovy.lang.Closure.call(Closure.java:418)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:101)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.lang.NullPointerException: Cannot invoke method isAnnotationPresent() on null object
	at geb.spock.SpockGebTestManagerBuilder$_closure1.doCall(SpockGebTestManagerBuilder.groovy:25)
	at groovy.lang.Closure.call(Closure.java:418)
	at geb.test.GebTestManager.getResetBrowserAfterEachTest(GebTestManager.groovy:163)
	at geb.test.GebTestManager.afterTestClass(GebTestManager.groovy:125)
	at geb.spock.GebExtension.addManagerCalls_closure2(GebExtension.groovy:64)
	at groovy.lang.Closure.call(Closure.java:418)
	at org.spockframework.runtime.extension.MethodInvocation.proceed(MethodInvocation.java:101)
	at groovy.lang.Closure.call(Closure.java:418)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.run(Closure.java:500)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Seems like some kind of unexpected Geb issue though. Not necessarily Spock related. I might be able to simply ignore this execution exception...

[Vampire]

The problem is, that Geb records a stack of test classes in thread locals.
It puts this onto the stack in a shared initializer interceptor (before you change the thread with that extension), but tries to get it in an iteration interceptor where you changed the thread and thus nothing is on the thread-local stack.

The problem is, that where the NPE is happening, Geb would then reset the browser after each test unless you used @Stepwise on the specification and this fails due to the NPE happening, eventually also skipping other code of other extensions.

With Spock 2.4 Geb will be able to use the new store feature to properly store information instead of using thread-locals.
To work-around this new problem I guess one way would be to push to that stack before you do the invoke call in the work-around.
But actually the beforeTestClass is doing more and also other thread-local things, so maybe you need to re-call that method unless it does other things that would then be bad, I don't know.

[Vampire]

Another idea how to maybe approach your original problem.
Indeed build such a resource pool.
Have a Semaphore with parallelism amount of permits.
acquire it before you create the driver, release it after you closed the driver.
This way max 5 drivers are used as the others are waiting for the semaphore to have a permit available.
This might be the best solution in your case.

[roded]

That's interesting, but I'm not sure though where I should acquire and release the Semaphore permits.
If I do it in the spec's setupSpec and cleanupSpec could I be risking a deadlock somehow?
Is there a nice way of doing it in an extension?

The following seems to be behaving as required, do you see any issue with it?

    @Override
    void visitSpec(SpecInfo spec) {
        spec.addInterceptor {
            semaphore.acquire()
            it.proceed()
            semaphore.release()
        }
    }

[Vampire]

You should probably release in a finally so that the semaphore is released even if there is an exception or test failure, but other than that, I think that should be appropriate, yes.

[Vampire]

Given you want each and every spec in that pool.
Otherwise you could do it via annotation-driven extension and mark the specifications that should take part explicitly or could even declare several pools if necessary.

[roded]

finally: yes, absolutely.
I thinks this works out well for my purposes. I shall try perusing it further. So far, it seems to be working quite well.
At the moment, I need to apply it to all specs, but I understand the solution you're suggesting for applying it for a subset of specs.

Thanks so much for the attention and help @Vampire, this discussion was very very helpful.