I apply a function to the first axis of the input Pandas Dataframe which does the extraction. My script extracts titles, surnames, given names and guess biological sex and martial status. I also have a dataframe in the external script containing valid first names and associated genders parsed from a JSON which is passed as a nvarchar(max). The temp table providing the input has different columns for the actual-, birth- and mother names for different people. I am only allowed to share a limited amount: I work on SQL Server 2019 Development version we also have an enterprise version which I guess doesn't matter here. Or perhaps there is a better way to parallel execution of an exterenal script? What if I horizontally partition the temp table which holds the input data and execute the external script on the partitions using asynchronous procedure execution, then I just get back their union? Python's multiprocessing module doesn't work inside the external script. I already heavily optimized the script, but it is still slow. I wrote the script it works but the input data is extremely large so the external script execution takes a long time.
So I must use a R or Python external script. I have to write a query which must rely on regular expressions and I am not allowed to use CLR or use anything outside the SQL Server environment. I am not allowed to do the above, because of workspace/interpersonal reasons.
Normally you should not rely on external scripts to achieve high performance. If this is not the case use CLR or process your data in a external application where the functionality is present. If your external script is too slow, you may do something in your scritpt which can be implemented in TSQL, which is a big problem.
0 kommentar(er)
