[chaen]

BEGINRELEASENOTES

*WMS

CHANGE: select pilot info in a single query instead of a large IN list and 2 queries

ENDRELEASENOTES

[aldbr]

I am not fully sure to understand what you are trying to solve exactly here, can you provide further details please?

I assume it's because of:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/PilotStatusAgent.py

Lines 85 to 98 in 42c75a1

	result = self.pilotDB.selectPilots(
	{"Status": PilotStatus.PILOT_TRANSIENT_STATES}, older=timeLimitToConsider, timeStamp="LastUpdateTime"
	)
	if not result["OK"]:
	self.log.error("Failed to get the Pilot Agents")
	return result
	if not result["Value"]:
	return S_OK()
	refList = result["Value"]
	result = self.pilotDB.getPilotInfo(refList)
	if not result["OK"]:
	self.log.error("Failed to get Info for Pilot Agents")
	return result

If it's the case, I am wondering why we don't just simplify by having a getPilotInfoForSelection method (merging selectPilots and getPilotInfo into a single SQL request). Or even having selectPilots returning any pilot information? (that would make a breaking change in the API here)

[chaen]

the problem is the query itself.

select * from Table where column in (1,2,3,4,...,126456464)

is not efficient.

On the other hand, this is very fast

create temporary table tmp_table(column INT PRIMARY KEY);
insert into tmp_table (column) values (1),(2),(3),(4),...,(126456464);
select * from table t join temporary table tt on t.column = tt.column;

Of course, it's for large number of values.

[aldbr]

the problem is the query itself.

select * from Table where column in (1,2,3,4,...,126456464)

is not efficient.

This I understand, but it looks like the query itself could be avoided by returning the pilot info instead of just the pilot reference from selectPilots(), unless I am missing some details

[fstagni]

the problem is the query itself.

select * from Table where column in (1,2,3,4,...,126456464)

is not efficient.

This I understand, but it looks like the query itself could be avoided by returning the pilot info instead of just the pilot reference from selectPilots(), unless I am missing some details

Apart from the call from the web, for which it might be true what you are saying, the general issue would still be there, right? @chaen when did you observe this? Which was the caller?

[chrisburr]

IIRC the stalled job agent does a query like this?

[chaen]

the problems were the site directors. I am happy with any change honestly, this was just an emergency fix

[aldbr]

I have changed the PR to just avoid the large IN by having 1 SQL request instead of 2.
I have tested it in lhcbdirac production, I haven't encountered any issue.
From a brief overview, it seems to improve the duration of the monitoring cycles but it doesn't seem to be significant.

[fstagni]

Are you sure?
If that's what you wanted, not even a warning here?

[aldbr]

I adjusted the function to done a single query instead of 2, the error is now returned to the caller