Lazy loading of huge dataset

[maksymg]

How to work efficiently with huge dataset?
In JSP based app I have Prev, Next button functionality based on:
- client possibility to remember current position;
- business layer possibility to return n records start from m position

In Laszlo case it should be transfered in "Load More data" button functionality. But how to load additional data to already loaded dataset efficiently?

[johncorro]

I'm not sure I understand 100% of what you're looking for, but I think I got the basic idea. Sounds like you're looking for how to deal w/ pagination over a large number of records. Your goal is that you want to allow a user to iterate over a large result set, but only load the data for the records they are immediately viewing to improve performance.

I'd be interested in hearing other people's input on this, but the best approach I've seen to this is do the following:

- When the user submits a request to initially generate the listing, the DAO layer performs 2 queries. The first query performs a search for all the appropriate records, but only returns the *Primary Keys*. This list is expected to be cached at a higher level (either the service layer or the UI layer). The second query performs a search for more meaningful data based on the primary keys that are passed to it.

- As the user moves from page to page, the appropriate number of primary keys are passed down to the DAO layer.

There's some more refinement going on when you put it into actual usage, but this gives you a rough idea of the approach.

[maksymg]

Let's say you have a table with a lot of records. Your interface gives you a possibility to sort this records. And you want to show some first of them to user. In JSP based application the system just shows 1 - n of m records. If user clicks next - it will show n - n+n of m records. And be aware nobody keeps cache of the records or primary keys in memory it's unacceptable. The system just rerun the query and skip some first records. It's easy to build, since I recreate the page on a server for each request. For Laszlo client everything works fine if you able to load all data at once. So I'm looking for a possibillity to extend loaded into Laszlo client data lazily?? Any ideas?

[johncorro]

In my experience, if the DAO has to be smart enough to retrieve only a certain range of records within a result set, the SQL tends to become database-specific. That's not necessarily an issue to everyone. I'm assuming it's not in this case.

In any event, I'm afraid I don't understand what you're looking to do...

You indicated that you want to be able to handle pagination over a large dataset in a lazy loaded manner. If your DAO layer is capable of retrieving an arbitrary range of records within a query, then what more do you have to do? That's about as lazy loaded as it gets. The only thing you'd have to do on the LZX side is maintain the indexes of the current records. On the "Next" or "Previous" buttons, you'd have to provide the index range you'd want to retrieve. When the user presses either button, you simply invoke the approrpriate JSP page (if you're using HTTP) or the appropriate method (if you're using RPC) and render the results.

Are you looking to cache the records that have already been retrieved so that a server call doesn't have to be made again if the user goes back to that range of records? Perhaps you're making the confusion that an LZX view has to be reloaded (by the browser) whenever the user would press the "Previous" or "Next" button.

[maksymg]

Yes, I'm looking for cache possibility for Laszlo client. Let's say client is started and client grid is loaded with first 10 records. When user press "Next", I want to see 20 records in the grid. But Client must recieve form the server only new 10 records.

[maksymg]

Continue. My server side is able to create XML data that contain new records, but how I can combine this new portion with existing data on client side?

[johncorro]

Hmmm...that's a tough one. I'm afraid I don't know right off the bat how to do that.

One thing to consider though, if someone traverses to the very end of the resultset using your approach, they'll end up caching the entire dataset on their LZX app. You mentioned earlier that it was unacceptable to store the PKs on the client side - if it's unacceptable to store a collection of...let's say int...PK values on the client side, then there's no way you'll want to go forward w/ your approach of caching the record data as the user traverses the resultset.

[maksymg]

Well it's extreme version. It's not a business case. Most user are happy, let's say, with first 50 records. But some times they need first 200. So from an optimization stand point I don't want to load first 200 records for everybody to cover this rare cases.