partitioned-batch-job raises NPE when used with mutiple Partiton handlers #793
Comments
|
Thanks for writing the issue as also faced similar issue Thought to add more info and hope it may help to dig up This NPE will also be caused if the partitionHandler is at Below is a stackoverflow article created with sample code on each attempt Summary of issue Spring batch job 1 (received job parameter for setting for step 1/setting for step 2) The reason we want is to have different step with different k8s setting (like cpu/memory/grid) Attempts:
Complete Project can be found in The main class of configuration is try to simplify into one class
Complete Project can be found in The main class of configuration is try to simplify into one class Both Result Following (full trace at above git repo): During job trigger, it will error (it seem pass initial start up, but error during execution) Because below will only occur when there are multiple Full Log Study/Reference: Thanks for info/helps in advance |
|
The symptom seem to be the In database, what we see is all null (56 is the one map to batch job, and 57 is the worker execution id base on above reproducing code) If PartitionHandler is deploy as My current hypothesis is f more than one |
|
Hi, just a short note to mention that, in my case, this same behaviour appears while the partition handlers are annotated with @bean. However, if the partition handler is annotated with @bean and @StepScope, then the exception raised is as per Issue #792. In this last case, even if there is only one partition handler and if it is annotated with @StepScope, the exception is raised. |
|
After digging more, identify a theory and a workaround (but may need others to confirm, after testing not found any issue) Root cause analyze (hypothesis)The problem is DeployerPartitionHandler utilize annoation @BeforeTask to force task to pass in TaskExecution object as part of Task setup But as this partionerHandler is now at @StepScope (instead of directly at @bean level with @enable Task) or there are two partitionHandler, that setup is no longer triggered, as Resulted created DeployerHandler faced a null with Workaround ResolutionBelow is essentially a workaround to use the current job execution id to retrieve the associated task execution id Full code can be found in https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope_workaround_resolution In the partitionHandler method |
|
Hi, thanks for your comments. I'm not sure what do you exactly mean by: |
|
Hello, Sorry post wrong, but above should work for both What I mean above is,
|
|
Hello, In the initial issue I've registered (#791) I was wondering how could I avoid hard coding the step name into the partitionHandler() method ? For the records, the current example does that: So, my question was how to avoid hard-coding the setp name (here "workerStep") ? Looking for an answer to that question, I thought I'll be adding an additional parameter of type String, containing the step name. This way, the partitionHandler() methos would have become: And in order to initialize the stepName I would have had: And in order to be able to pass the step name from the step definition, I needed to decorate the partitionHandler method with @ScopeStep. Then, I experienced the exception described in the 2nd issue (#792). The next attempt I tried was to have several dedicated partition handlers, each one hard-coding the name of the associated partitioned worker step. This didn't require any more to annotate the partition handler methods with @StepScope, but I got the the case of the NPE, as described in the issue #793. Now, I tested your workaround and I think it works. So the partitionHandler() method loks now as follows: Running it, I don't have any more neither the NPE, nore the java.lang.IllegalStateException: No context holder available for step scope one. The commands are being started now, as the log file shows it: However, the command execution fails: Looking in the log files I got: Not sure whether this exception has anything to do with your workaround or if it's just an error of mine somewhere, I need to double check. Could you please confirm that it is not raised in your case ? Also, could you please suggest a recommended way such that to avoid hard-coding the step name ? Would it be better to have only a partition handler bean taking as an input argument of its constructor the step name ? And if yes, what would be the best way to do it ? Or would it be better to have as many dedicated partition handler beans as partitioned worker steps, each one hard-coding the step name ? Many thanks in advance. |
|
Hello, Sorry will need to answer question in reverse order as it may explain better. I actually faced that above issue with So it is why I ended up with only 1 partitionerhandler + 1 splitter and utilize Actually after thinking about it, I may not have a solution for this #793, but rather suggest to use #792 but with this workaround + stepExecution to decorate the partitioner handler on different use case. This also answer your first question about how to pass in step dynamically (sort of) by I am using stepExecutionContext -> find out the trigger step name (from your example, one step is You can see the full config in PartitionerHandler (note it utilize Partitioner (note it utilize Worker steps define, notice there are three steps Job setup Below can be in job parameters (so then can configure to use pass in, I just used job execution) Full class implementation |
|
Hello, Your answers as well as your suggestions are higly appreciated, thank you very much. In order to keep you updated, after having followed your workaround and your sample code, I finished up with the following partition handler: So, there is only one partition handler annotated with @beans and @StepScope. Running that doesn't unfortunatelly work in my case and it raises the following exception: ... This happens in this line of code: Setting a breakpoint there I can see that the execution context returned by stepExecution.getJobExecution().getExecutionContext Here the argument passed as stepExecution is null which is probably why the execution context is empty. Then I thought to modify the partition handler as follows: } Hence to pass the step name as an input argument of the partition handler instead of taking it from the execution context. Accordingly, the step definition invoking the partition handler looks like this: Now, running it the previous exception is gone, the commands are started but, after a while, all the commands return I have committed my small project here: https://github.com/nicolasduminil/partitioned-job.git. It's essentially the original example slightly modified in order to apply your work-around. I'd be very obligated to you if you have time to have a look at it and to let me know if you see something. Many thanks in advance for enlightening me and clarify this point. |
|
Last but not least, if someone from this code maintainers group or team could provide some guidance such that to get the code working, it would be great ! |
|
Thank you both for your comments. I was able to apply the workaround provided by @danilko and very much appreciate your effort on this. To get around the ClassCastException, I replaced the line with my own step-name specific to the step-scope. My use case is a job with multiple steps, where each step is remote-partitioned, and each step may have completely different This is a real issue and also request the code maintainers to address this issue with a more formal fix. |
|
In this case it seems we need to create a story for Spring Cloud Task to support multiple partition handlers. |
|
@cppwfs : which means that currently Spring Cloud Task isn't supposed to support multiple partition handlers ? I didn't see any note about that in the documentation. And what would be the idea of supporting only one partition handler ? Is there any assumption that a job or a flow only has to support one partitoned step ? Because since the annotating the partition handler with @StepScope, such that to be able to pass arguments from steps, isn't supported neither, having several step dedicated partition handlers is the only solution. Right ? |
|
@nicolasduminil Sorry for delay, was recently being very busy with work related tasks. I did look your code and create a branch under your repo after my suggestion fixes (after these fixes it is working for me in my local setup) Into complete state and can see four java dynamic process spawn up during the run Following are list of changes I made within that branch The first one with partitionhandler, in my test, actually you will not need to specify the jobexplorer and others as it is already autowired with bean, so I simplify a lot to only Regarding step name, in my tests/my previous setups, will not recommend straight value (without bean or context) as it seem when system created them, they will act strange as bean and cause null or strange That job context seem above is populated by adding a listener to job Then in that jobExecutionListner, populate this relation, so can be used in partitionhandler (the other way is to use jobparameter or some sort of DB) And I think the most important (but not 100% sure if my analyze is correct), I always need to ensure max worker is great than grid size, otherwise will face the problem with step (which kind make sense, consider that how to split grid to worker when worker is less than grid) |
|
@danilko: Many thanks again for your help and support. Your contribution has been highly appreciated. Many thanks also for having dramatically simplified the provided code and for having made it work. |
Hello,
I modified the partitioned-batch-job sample such that to add the following method:
Running the job raises the following exception:
Please advise.
Many thanks in advance.
The text was updated successfully, but these errors were encountered: