ipcSkillProcess and results buffering

Just to add a little detail - I think the size of the chunk that's read is a feature of the linux call used to read from the pipe and not a feature of Virtuoso though there could also be chunking going on on the Virtuoso side, not sure.

The particularly annoying issue for me is that the simple and clear code mostly works fine in practice.  The problem is that it doesn't work in theory (!) - there is a potential future bug lurking when a result size exceeds the (unknown) chunk size limit.   It seems a pity to drastically complicate my code for this unlikely scenario but for safety I think I need to.  Or at least document the limit.

BS202501273649 said:

I think the size of the chunk that's read is a feature of the linux call used to read from the pipe

It's not quite that simple, but either way it's an implementation detail because of not knowing how much is coming from the interface in advance.

ipcReadProcess does document the 4096 byte buffer, but from some experiments I also see it double that sometimes. I can see in the implementation there are some buffers which are double (excluding termination).

As Aurel said, the standard way is to put the SKILL into a file, and then have the python program send the SKILL to load the file. That avoids the issue (I think Aurel's answer was for the direction you're using).

Andrew

BS202501273649 said:

I think the size of the chunk that's read is a feature of the linux call used to read from the pipe

It's not quite that simple, but either way it's an implementation detail because of not knowing how much is coming from the interface in advance.

ipcReadProcess does document the 4096 byte buffer, but from some experiments I also see it double that sometimes. I can see in the implementation there are some buffers which are double (excluding termination).

As Aurel said, the standard way is to put the SKILL into a file, and then have the python program send the SKILL to load the file. That avoids the issue (I think Aurel's answer was for the direction you're using).

Andrew

I've changed the read code so it carries on reading until the bytes represent a complete S expression.  That works in practice and (I think) in theory because the result of evaluating the SKILL is definitely going to be an S expression I think.   Anyway it's passing my tests even when the result of the SKILL is a really long thing, and no file reading so it still performs well.   Seems OK for now but I'll bear in mind the idea of going via a file. Seems ugly but maybe it makes sense.

Do you have an example of how a long SKILL string would be passed from Python to the ipcSkillProcess interpreter?

(what kind of string and how it's generated)

When I said to put the data coming from Python into a file, I had in mind any generated data, not specifically SKILL code.

In my experience it is often list of points or polygons that are processed by external programs to be highlighted inside Virtuoso. Those could become relatively big and might break the interpretation limit when passed directly, this is where using a separated file solves the issue.

By any chance, are you using an LLM to generate SKILL code from Python and giving its output to ipcSkillProcess interpreter? If so it would make a lot of sense to have this output inside a dedicated file that you can Lint and debug afterwards.

Otherwise, a good practice is to define in advance all the functions your external process will be able to call, and make it write simple calls to the ipcSkillProcess interpreter. That way you should never reach the limit.

(if this case does not apply, I'd be glad to understand how and why)

As I say, it's the return of the SKILL function that can be too long.  The SKILL function requested from the Python is (as you suggest) typically just a single call.  But it can return (in theory though generally not in practice) a very long expression.   For example, in testing, I call a function which returns a list of all the numbers from 1 to n.   Once n gets big enough, then my first read only gets the first chunk (typically 64k worth) and I need to read again to get the rest.  My changed code handles that - the first chunk is not a complete S expression so it reads again and gets the rest.

It would be nice to just require all the SKILL functions that can be called from the Python to return small results.  But this is intended to be a flexible middleware component so I can't a priori restrict the return values of the calls - I mean, I can, but it would make it less reliable and robust.

This solution seems to be OK for now so I'm going to carry on with that.