[atsareg]

The removed operation was reseting files to Unused for tasks that were created but job not yet submitted. As a result new jobs were created for the same files multiple files.

A fix was needed in the pre-commit configuration for the flynt part as by default it uses python3.14 version not compatible with 3.11. I think this should be harmless

BEGINRELEASENOTES

*Transformation
FIX: WorkflowTasks - do not reset files for not yet submitted jobs

ENDRELEASENOTES

[fstagni]

Looking at this PR, I realize that there is the "same" function in https://github.com/DIRACGrid/DIRAC/blob/rel-v8r0/src/DIRAC/TransformationSystem/Client/RequestTasks.py#L366

which is certainly working fine (in the context of replication transformations, but still...). Can you compare the 2 and check their differences?

[atsareg]

Yes, this is a good confirmation of this PR. In the replication transformations the files are ONLY checked for tasks that were created and got the ExternalID:

DIRAC/src/DIRAC/TransformationSystem/Client/RequestTasks.py

Line 389 in d20f5d9

# Only consider tasks that are submitted, ExternalID is a string

.

Here, there should be the same logic: files are reserved while the jobs are not yet submitted, those should not be reset to UNUSED because it is just a slow job submission in an overloaded system.

[fstagni]

@arrabito and @zhangxiaomei can you evaluate this one? Is it by chance something that you hotfixed?

[fstagni]

Thanks @atsareg, the comparison with RequestTasks.getSubmittedFileStatus() is convincing: only checking files of tasks that actually got an ExternalID is the right logic, and resetting reserved files to Unused on slow submission clearly causes duplicate job submissions.

Two things before we merge:

1. @arrabito @zhangxiaomei could you confirm whether this matches the hotfix you applied? (asking since Sept, following up here)
2. One edge case to confirm is accepted: with this change, files of tasks whose WMS jobs were deleted/purged from job monitoring are no longer recycled to Unused (previously the same code path did it). Those tasks still go to Failed, but their files would remain Assigned forever and never be reprocessed. Is this an accepted regression for you, or should we handle it (e.g. recycle files when the task is set to Failed for a removed job)?

Minor: the deletion leaves noTasks = res["Value"]["NoTasks"] unused in getSubmittedFileStatus - can you remove that line while at it?

[atsareg]

Deleting jobs in a non-final status - is there a use case for that for Transformation jobs ? This can be only done by hand, I think. If a Transformation is found to be wrongly defined, I can imagine that all of its jobs will be also deleted even in a non-final status. But the Transformation will be also deleted as well, so its file statuses are no relevant any more.

Normally the Production jobs are not purged until the corresponding Transformation is stopped AFAIU.
If a job is in the final status Done or Failed, the corresponding files get updated to Processed/Unused.

I have removed the redundant line